2024-03-24T10:58:48.865 INFO:root:teuthology version: 0.0.1.dev262+ge691533 2024-03-24T10:58:48.865 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_teuthology@teuthology --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --name teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi --archive /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869 -- /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869/orig.config.yaml 2024-03-24T10:58:48.919 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T10:58:49.014 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869 branch: squid description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: ceph-qa@ceph.com first_in_suite: false job_id: '7617869' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 9.stream overrides: admin_socket: branch: squid ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 8903 sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 sleep_before_teardown: 0 subset: 44/64 suite: orch suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: f73f1dfd96849300c4322bd7259a3ab37f94c200 targets: smithi100.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExVPUXHvqJ01g77jhqC9m8JF9H7OwrJ59FB7xuWe9abrrCuVBWbInMLNUiFeGmZXPIwC418IuxLihAZkbqS1hs= smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGXhGVBiLZy1w5q+/zMisSsFsc4cxVZ425x5JXIOp1EGE9jZgKW+OQIkf342Y8stnnUL9Mtjj+BBtD5OUguyDhc= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] variables: fail_fs: true teuthology_branch: main teuthology_sha1: e691533f9cbb33d85b2187bba20d7102f098636d timestamp: 2024-03-22_21:08:02 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done - ceph versions | jq -e '.mgr | length == 1' - ceph versions | jq -e '.mgr | keys' | grep $sha1 - ceph versions | jq -e '.overall | length == 2' - ceph orch upgrade check quay.ceph.io/ceph-ci/ceph:$sha1 | jq -e '.up_to_date | length == 2' - ceph orch ps - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-03-24T10:58:49.014 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa; will attempt to use it 2024-03-24T10:58:49.015 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks 2024-03-24T10:58:49.015 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-03-24T10:58:49.017 INFO:teuthology.task.internal:Checking packages... 2024-03-24T10:58:49.040 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'db0330b1e4e2470d52b750e251e55a522b4f7d69' 2024-03-24T10:58:49.040 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T10:58:49.040 INFO:teuthology.packaging:ref: None 2024-03-24T10:58:49.040 INFO:teuthology.packaging:tag: None 2024-03-24T10:58:49.040 INFO:teuthology.packaging:branch: squid 2024-03-24T10:58:49.041 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T10:58:49.041 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=squid 2024-03-24T10:58:49.261 INFO:teuthology.task.internal:Found packages for ceph version 19.0.0-1764.gf73f1dfd 2024-03-24T10:58:49.263 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-03-24T10:58:49.269 INFO:teuthology.task.internal:no buildpackages task found 2024-03-24T10:58:49.269 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-03-24T10:58:49.279 INFO:teuthology.task.internal:Saving configuration 2024-03-24T10:58:49.301 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-03-24T10:58:49.307 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T10:58:49.326 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:53:40.056296', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExVPUXHvqJ01g77jhqC9m8JF9H7OwrJ59FB7xuWe9abrrCuVBWbInMLNUiFeGmZXPIwC418IuxLihAZkbqS1hs='} 2024-03-24T10:58:49.348 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:53:40.054950', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGXhGVBiLZy1w5q+/zMisSsFsc4cxVZ425x5JXIOp1EGE9jZgKW+OQIkf342Y8stnnUL9Mtjj+BBtD5OUguyDhc='} 2024-03-24T10:58:49.349 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-03-24T10:58:49.354 INFO:teuthology.task.internal:roles: ubuntu@smithi100.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-03-24T10:58:49.354 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-03-24T10:58:49.354 INFO:teuthology.run_tasks:Running task console_log... 2024-03-24T10:58:49.467 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fb95b0710d0>, signals=[15]) 2024-03-24T10:58:49.467 INFO:teuthology.run_tasks:Running task internal.connect... 2024-03-24T10:58:49.473 INFO:teuthology.task.internal:Opening connections... 2024-03-24T10:58:49.473 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T10:58:49.475 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:58:49.564 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-03-24T10:58:49.565 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T10:58:49.654 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-03-24T10:58:49.660 DEBUG:teuthology.orchestra.run.smithi100:> uname -m 2024-03-24T10:58:49.677 INFO:teuthology.orchestra.run.smithi100.stdout:x86_64 2024-03-24T10:58:49.677 DEBUG:teuthology.orchestra.run.smithi100:> cat /etc/os-release 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:NAME="CentOS Stream" 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION="9" 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:ID="centos" 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:VERSION_ID="9" 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:58:49.737 INFO:teuthology.orchestra.run.smithi100.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:ANSI_COLOR="0;31" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:58:49.738 INFO:teuthology.orchestra.run.smithi100.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:58:49.739 INFO:teuthology.lock.ops:Updating smithi100.front.sepia.ceph.com on lock server 2024-03-24T10:58:49.756 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-03-24T10:58:49.774 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-03-24T10:58:49.775 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-03-24T10:58:49.831 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-03-24T10:58:49.832 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-03-24T10:58:49.833 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-03-24T10:58:49.848 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-03-24T10:58:49.856 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-03-24T10:58:49.930 INFO:teuthology.task.internal:Checking for old test directory... 2024-03-24T10:58:49.930 DEBUG:teuthology.orchestra.run.smithi100:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:58:49.933 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-03-24T10:58:49.948 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-03-24T10:58:49.969 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-03-24T10:58:49.969 DEBUG:teuthology.orchestra.run.smithi100:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:58:49.994 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-03-24T10:58:50.296 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-03-24T10:58:50.343 INFO:teuthology.run_tasks:Running task kernel... 2024-03-24T10:58:50.357 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-03-24T10:58:50.357 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-03-24T10:58:50.357 DEBUG:teuthology.orchestra.run.smithi100:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:58:50.358 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-03-24T10:58:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:58:50.374 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T10:58:50.375 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-03-24T10:58:50.375 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-03-24T10:58:50.432 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:58:50.432 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-361.el9.x86_64 2024-03-24T10:58:50.432 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-03-24T10:58:50.435 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-361.el9.x86_64 2024-03-24T10:58:50.436 INFO:teuthology.task.kernel:Running kernel on smithi100: 5.14.0-361.el9.x86_64 2024-03-24T10:58:50.436 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum install -y kernel 2024-03-24T10:58:52.498 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - AppStream 15 MB/s | 6.7 MB 00:00 2024-03-24T10:58:52.525 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 14 MB/s | 6.7 MB 00:00 2024-03-24T10:58:53.890 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - BaseOS 7.2 MB/s | 2.0 MB 00:00 2024-03-24T10:58:53.942 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 6.5 MB/s | 2.0 MB 00:00 2024-03-24T10:58:54.983 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.0 MB 00:00 2024-03-24T10:58:56.892 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 22 MB/s | 19 MB 00:00 2024-03-24T10:59:00.809 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 13 MB/s | 6.1 MB 00:00 2024-03-24T10:59:02.369 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 16 kB 00:00 2024-03-24T10:59:02.813 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 958 kB/s | 8.0 MB 00:08 2024-03-24T10:59:03.978 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 14 MB/s | 21 MB 00:01 2024-03-24T10:59:04.981 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 17 MB/s | 19 MB 00:01 2024-03-24T10:59:08.928 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T10:59:09.231 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 7.9 MB/s | 6.1 MB 00:00 2024-03-24T10:59:11.043 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:59:11.044 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:59:11.141 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:59:11.152 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:59:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:59:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T10:59:11.153 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T10:59:11.153 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T10:59:11.153 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-03-24T10:59:11.153 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T10:59:11.154 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 76 M 2024-03-24T10:59:11.154 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 123 M 2024-03-24T10:59:11.154 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T10:59:11.288 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 486 kB/s | 44 kB 00:00 2024-03-24T10:59:11.524 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 13 kB/s | 16 kB 00:01 2024-03-24T10:59:11.788 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 25 MB/s | 15 MB 00:00 2024-03-24T10:59:12.097 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 37 MB/s | 33 MB 00:00 2024-03-24T10:59:12.347 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 26 MB/s | 27 MB 00:01 2024-03-24T10:59:12.347 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:59:12.348 INFO:teuthology.orchestra.run.smithi105.stdout:Total 64 MB/s | 76 MB 00:01 2024-03-24T10:59:12.460 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-24T10:59:13.012 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T10:59:13.377 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T10:59:13.377 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T10:59:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T10:59:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T10:59:16.851 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T10:59:17.453 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 14 kB/s | 1.7 kB 00:00 2024-03-24T10:59:17.469 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:59:17.572 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:59:19.565 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-03-24T10:59:19.565 INFO:teuthology.orchestra.run.smithi100.stdout:Package kernel-5.14.0-361.el9.x86_64 is already installed. 2024-03-24T10:59:19.662 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout: kernel x86_64 5.14.0-430.el9 baseos 44 k 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T10:59:19.667 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core x86_64 5.14.0-430.el9 baseos 15 M 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules x86_64 5.14.0-430.el9 baseos 33 M 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core x86_64 5.14.0-430.el9 baseos 27 M 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:Install 4 Packages 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T10:59:19.668 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 76 M 2024-03-24T10:59:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 123 M 2024-03-24T10:59:19.669 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T10:59:19.810 INFO:teuthology.orchestra.run.smithi100.stdout:(1/4): kernel-5.14.0-430.el9.x86_64.rpm 442 kB/s | 44 kB 00:00 2024-03-24T10:59:20.144 INFO:teuthology.orchestra.run.smithi100.stdout:(2/4): kernel-core-5.14.0-430.el9.x86_64.rpm 34 MB/s | 15 MB 00:00 2024-03-24T10:59:20.349 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:59:20.483 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:59:20.552 INFO:teuthology.orchestra.run.smithi100.stdout:(3/4): kernel-modules-5.14.0-430.el9.x86_64.rpm 40 MB/s | 33 MB 00:00 2024-03-24T10:59:20.803 INFO:teuthology.orchestra.run.smithi100.stdout:(4/4): kernel-modules-core-5.14.0-430.el9.x86_6 28 MB/s | 27 MB 00:00 2024-03-24T10:59:20.803 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T10:59:20.803 INFO:teuthology.orchestra.run.smithi100.stdout:Total 67 MB/s | 76 MB 00:01 2024-03-24T10:59:21.462 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T10:59:21.809 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T10:59:21.809 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T10:59:22.831 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T10:59:22.831 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T10:59:25.166 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:59:25.283 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:25.354 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T10:59:26.033 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-core-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:59:26.128 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:59:28.855 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:59:28.999 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:59:30.011 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:33.728 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T10:59:34.066 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:38.805 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:59.532 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:59.533 INFO:teuthology.orchestra.run.smithi105.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T10:59:59.533 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T10:59:59.577 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:59.969 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T10:59:59.969 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T10:59:59.969 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T10:59:59.969 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:00:00.377 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:00:00.378 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:00:00.704 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:00:00.850 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:00:00.850 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:00:00.850 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:00:00.850 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:00:00.850 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:00:00.851 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-03-24T11:00:00.851 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:00:00.851 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:00:00.930 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:00:00.930 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:00:00.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.06 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 68.el9 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Group : Unspecified 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 8264254 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:00:00.977 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : builder@centos.org 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:00:00.978 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:00:00.979 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:00:00.979 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:00:00.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:00:01.576 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-03-24T11:00:02.962 INFO:teuthology.orchestra.run.smithi105.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:00:02.992 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-03-24T11:00:02.995 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-03-24T11:00:03.010 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.7aVv4MAnx9 2024-03-24T11:00:03.010 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.7aVv4MAnx9 2024-03-24T11:00:03.079 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.7aVv4MAnx9 2024-03-24T11:00:03.245 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.7aVv4MAnx9 is 6KB 2024-03-24T11:00:03.293 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.7aVv4MAnx9 2024-03-24T11:00:03.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:00:03.381 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:00:03.381 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:00:03.381 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:00:03.381 INFO:teuthology.orchestra.run.smithi105.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:00:03.383 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:00:03.523 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-03-24T11:00:08.282 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:00:08.282 INFO:teuthology.orchestra.run.smithi100.stdout:dracut: Disabling early microcode, because kernel does not support it. CONFIG_MICROCODE_[AMD|INTEL]!=y 2024-03-24T11:00:08.282 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:00:08.326 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-modules-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:00:08.711 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: kernel-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:00:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-5.14.0-430.el9.x86_64 1/4 2024-03-24T11:00:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-core-5.14.0-430.el9.x86_64 2/4 2024-03-24T11:00:08.712 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-5.14.0-430.el9.x86_64 3/4 2024-03-24T11:00:09.151 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : kernel-modules-core-5.14.0-430.el9.x86_64 4/4 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-5.14.0-430.el9.x86_64 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-core-5.14.0-430.el9.x86_64 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-5.14.0-430.el9.x86_64 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout: kernel-modules-core-5.14.0-430.el9.x86_64 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:00:09.152 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:00:09.498 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:00:09.591 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:00:09.591 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:00:09.591 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-361.el9.x86_64 Expected: 5.14.0-430.el9.x86_64 2024-03-24T11:00:09.591 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-03-24T11:00:09.591 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-03-24T11:00:09.592 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-03-24T11:00:09.592 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi100.front.sepia.ceph.com, path=None, version=distro) 2024-03-24T11:00:09.592 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q kernel | sort -rV | head -n 1 2024-03-24T11:00:09.671 INFO:teuthology.orchestra.run.smithi100.stdout:kernel-5.14.0-430.el9.x86_64 2024-03-24T11:00:09.672 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-430.el9.x86_64 2024-03-24T11:00:09.672 DEBUG:teuthology.orchestra.run.smithi100:> sudo rpm -qi grub2-tools 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Name : grub2-tools 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Epoch : 1 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Version : 2.06 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Release : 68.el9 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Architecture: x86_64 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Install Date: Fri 01 Sep 2023 09:14:30 PM UTC 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Group : Unspecified 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Size : 8264254 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:License : GPLv3+ 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-03-24T11:00:09.706 INFO:teuthology.orchestra.run.smithi100.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:Packager : builder@centos.org 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:Vendor : CentOS 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:URL : http://www.gnu.org/software/grub/ 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:Summary : Support tools for GRUB. 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:Description : 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:customizable bootloader with modular architecture. It supports a rich 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:variety of kernel formats, file systems, computer architectures and 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:hardware devices. 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:00:09.707 INFO:teuthology.orchestra.run.smithi100.stdout:This subpackage provides tools for support of all platforms. 2024-03-24T11:00:09.708 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-03-24T11:00:09.708 INFO:teuthology.task.kernel:Updating grub on smithi100 to boot 5.14.0-430.el9.x86_64 2024-03-24T11:00:09.708 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-03-24T11:00:10.338 INFO:teuthology.orchestra.run.smithi100.stderr:Generating grub configuration file ... 2024-03-24T11:00:11.705 INFO:teuthology.orchestra.run.smithi100.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-03-24T11:00:11.747 INFO:teuthology.orchestra.run.smithi100.stderr:done 2024-03-24T11:00:11.750 DEBUG:teuthology.orchestra.run.smithi100:> mktemp 2024-03-24T11:00:11.767 INFO:teuthology.orchestra.run.smithi100.stdout:/tmp/tmp.VaErQ1zNaG 2024-03-24T11:00:11.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.VaErQ1zNaG 2024-03-24T11:00:11.835 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0666 /tmp/tmp.VaErQ1zNaG 2024-03-24T11:00:11.971 DEBUG:teuthology.orchestra.remote:smithi100:/tmp/tmp.VaErQ1zNaG is 6KB 2024-03-24T11:00:12.019 DEBUG:teuthology.orchestra.run.smithi100:> rm -fr /tmp/tmp.VaErQ1zNaG 2024-03-24T11:00:12.034 DEBUG:teuthology.orchestra.run.smithi100:> sudo /bin/ls /boot/loader/entries || true 2024-03-24T11:00:12.098 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-0-rescue.conf 2024-03-24T11:00:12.098 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-239.el9.x86_64.conf 2024-03-24T11:00:12.098 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-361.el9.x86_64.conf 2024-03-24T11:00:12.098 INFO:teuthology.orchestra.run.smithi100.stdout:15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64.conf 2024-03-24T11:00:12.100 DEBUG:teuthology.orchestra.run.smithi100:> sudo grub2-set-default 15949e3c88704300882482eddfd7baae-5.14.0-430.el9.x86_64 2024-03-24T11:00:12.249 DEBUG:teuthology.orchestra.run.smithi100:> sudo shutdown -r now 2024-03-24T11:00:33.546 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-03-24T11:00:33.547 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:00:33.548 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:00:42.260 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-03-24T11:00:42.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-24T11:00:42.261 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:01:16.959 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-03-24T11:01:25.966 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-24T11:01:25.967 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:01:33.556 DEBUG:teuthology.orchestra.remote:timed out 2024-03-24T11:01:42.562 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:01:42.563 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:01:42.905 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-03-24T11:01:43.303 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:01:43.303 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:01:43.304 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-03-24T11:01:43.318 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:01:43.318 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:01:43.318 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:01:43.318 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:01:44.319 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-03-24T11:01:44.319 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:01:44.354 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.100 2024-03-24T11:01:44.383 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-03-24T11:01:44.421 DEBUG:teuthology.parallel:result is None 2024-03-24T11:01:56.355 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-24T11:01:56.356 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:01:56.932 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-03-24T11:01:57.353 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-24T11:01:57.353 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-430.el9.x86_64"... 2024-03-24T11:01:57.353 DEBUG:teuthology.orchestra.run.smithi100:> uname -r 2024-03-24T11:01:57.367 INFO:teuthology.orchestra.run.smithi100.stdout:5.14.0-430.el9.x86_64 2024-03-24T11:01:57.367 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-430.el9.x86_64 vs 5.14.0-430.el9.x86_64 2024-03-24T11:01:57.367 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-03-24T11:01:57.367 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-03-24T11:01:58.368 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-03-24T11:01:58.368 DEBUG:teuthology.orchestra.run.smithi100:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-03-24T11:01:58.430 INFO:teuthology.orchestra.run.smithi100.stdout:ttyS1 2024-03-24T11:01:58.468 DEBUG:teuthology.parallel:result is None 2024-03-24T11:01:58.468 INFO:teuthology.run_tasks:Running task internal.base... 2024-03-24T11:01:58.478 INFO:teuthology.task.internal:Creating test directory... 2024-03-24T11:01:58.478 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:01:58.481 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-03-24T11:01:58.499 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-03-24T11:01:58.505 INFO:teuthology.run_tasks:Running task internal.archive... 2024-03-24T11:01:58.554 INFO:teuthology.task.internal:Creating archive directory... 2024-03-24T11:01:58.554 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:01:58.557 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-03-24T11:01:58.655 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-03-24T11:01:58.662 INFO:teuthology.task.internal:Enabling coredump saving... 2024-03-24T11:01:58.662 DEBUG:teuthology.orchestra.run.smithi100:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:01:58.664 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-03-24T11:01:58.697 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:01:58.706 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:01:58.711 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:01:58.719 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-03-24T11:01:58.721 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-03-24T11:01:58.728 INFO:teuthology.task.internal:Configuring sudo... 2024-03-24T11:01:58.728 DEBUG:teuthology.orchestra.run.smithi100:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:01:58.730 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-03-24T11:01:58.783 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-03-24T11:01:58.793 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-03-24T11:01:58.793 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:01:58.797 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-03-24T11:01:58.839 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:01:58.893 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:01:58.943 DEBUG:teuthology.orchestra.run.smithi100:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:01:59.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:01:59.067 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:01:59.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:01:59.136 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:01:59.181 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T11:01:59.228 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:01:59.310 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T11:01:59.356 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:01:59.356 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-03-24T11:01:59.424 DEBUG:teuthology.orchestra.run.smithi100:> sudo service rsyslog restart 2024-03-24T11:01:59.426 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-03-24T11:01:59.472 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:01:59.511 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:01:59.837 INFO:teuthology.run_tasks:Running task internal.timer... 2024-03-24T11:01:59.845 INFO:teuthology.task.internal:Starting timer... 2024-03-24T11:01:59.845 INFO:teuthology.run_tasks:Running task pcp... 2024-03-24T11:01:59.884 INFO:teuthology.run_tasks:Running task selinux... 2024-03-24T11:01:59.924 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-03-24T11:01:59.925 DEBUG:teuthology.orchestra.run.smithi100:> sudo service auditd rotate 2024-03-24T11:01:59.994 INFO:teuthology.orchestra.run.smithi100.stdout:Rotating logs: 2024-03-24T11:01:59.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-03-24T11:02:00.057 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-03-24T11:02:00.059 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-03-24T11:02:00.059 DEBUG:teuthology.orchestra.run.smithi100:> /usr/sbin/getenforce 2024-03-24T11:02:00.082 INFO:teuthology.orchestra.run.smithi100.stdout:Permissive 2024-03-24T11:02:00.082 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-03-24T11:02:00.106 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-03-24T11:02:00.106 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi100.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi105.front.sepia.ceph.com': 'permissive'} 2024-03-24T11:02:00.106 DEBUG:teuthology.orchestra.run.smithi100:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:02:00.153 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:02:00.154 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-03-24T11:02:00.182 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:02:00.182 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-03-24T11:02:00.183 DEBUG:teuthology.orchestra.run.smithi100:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:02:00.234 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-03-24T11:02:00.267 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-03-24T11:02:00.276 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-03-24T11:02:00.281 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-03-24T11:02:00.401 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-03-24T11:02:00.417 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-03-24T11:02:00.418 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi100.front.sepia.ceph.com,smithi105.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-03-24T11:06:15.573 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi100.front.sepia.ceph.com'), Remote(name='ubuntu@smithi105.front.sepia.ceph.com')] 2024-03-24T11:06:15.575 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-24T11:06:15.576 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi100.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:06:15.672 DEBUG:teuthology.orchestra.run.smithi100:> true 2024-03-24T11:06:15.749 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi100.front.sepia.ceph.com' 2024-03-24T11:06:15.749 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:06:15.750 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:06:15.846 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-03-24T11:06:15.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:06:15.927 INFO:teuthology.run_tasks:Running task clock... 2024-03-24T11:06:15.937 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-03-24T11:06:15.937 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:06:15.938 DEBUG:teuthology.orchestra.run.smithi100:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:06:15.940 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-03-24T11:06:15.941 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:06:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:06:15.986 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:06:16.004 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-03-24T11:06:16.014 INFO:teuthology.orchestra.run.smithi100.stderr:sudo: ntpd: command not found 2024-03-24T11:06:16.021 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-03-24T11:06:16.028 INFO:teuthology.orchestra.run.smithi100.stdout:506 Cannot talk to daemon 2024-03-24T11:06:16.044 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:06:16.052 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-03-24T11:06:16.060 INFO:teuthology.orchestra.run.smithi100.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:06:16.065 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-03-24T11:06:16.082 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-03-24T11:06:16.099 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-03-24T11:06:16.115 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:06:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:06:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-03-24T11:06:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.118 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.150 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:06:16.153 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:06:16.153 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-03-24T11:06:16.153 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.153 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.153 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.153 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:06:16.154 INFO:teuthology.run_tasks:Running task install... 2024-03-24T11:06:16.163 DEBUG:teuthology.task.install:project ceph 2024-03-24T11:06:16.164 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'}} 2024-03-24T11:06:16.164 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:06:16.164 INFO:teuthology.task.install:Using flavor: default 2024-03-24T11:06:16.170 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-03-24T11:06:16.170 INFO:teuthology.task.install:extra packages: [] 2024-03-24T11:06:16.171 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:06:16.171 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:06:16.171 INFO:teuthology.packaging:ref: None 2024-03-24T11:06:16.171 INFO:teuthology.packaging:tag: None 2024-03-24T11:06:16.171 INFO:teuthology.packaging:branch: quincy 2024-03-24T11:06:16.171 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:06:16.172 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-24T11:06:16.174 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69', 'tag': None, 'wait_for_package': False} 2024-03-24T11:06:16.174 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:06:16.174 INFO:teuthology.packaging:ref: None 2024-03-24T11:06:16.174 INFO:teuthology.packaging:tag: None 2024-03-24T11:06:16.174 INFO:teuthology.packaging:branch: quincy 2024-03-24T11:06:16.174 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:06:16.174 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-24T11:06:16.427 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/ 2024-03-24T11:06:16.427 INFO:teuthology.task.install.rpm:Package version is 17.2.7-726.g9c4e0c34 2024-03-24T11:06:16.480 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/ 2024-03-24T11:06:16.480 INFO:teuthology.task.install.rpm:Package version is 17.2.7-726.g9c4e0c34 2024-03-24T11:06:16.539 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:06:16.540 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:06:16.540 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:06:16.568 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:06:16.568 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:06:16.569 INFO:teuthology.packaging:ref: None 2024-03-24T11:06:16.569 INFO:teuthology.packaging:tag: None 2024-03-24T11:06:16.569 INFO:teuthology.packaging:branch: quincy 2024-03-24T11:06:16.569 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:06:16.569 DEBUG:teuthology.orchestra.run.smithi100:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:06:16.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:06:16.648 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-03-24T11:06:16.649 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:06:16.649 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-03-24T11:06:16.678 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-03-24T11:06:16.678 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:06:16.678 INFO:teuthology.packaging:ref: None 2024-03-24T11:06:16.678 INFO:teuthology.packaging:tag: None 2024-03-24T11:06:16.678 INFO:teuthology.packaging:branch: quincy 2024-03-24T11:06:16.678 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:06:16.678 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-03-24T11:06:16.719 DEBUG:teuthology.orchestra.run.smithi100:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:06:16.747 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-03-24T11:06:16.785 INFO:teuthology.orchestra.run.smithi100.stdout:check_obsoletes = 1 2024-03-24T11:06:16.787 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum clean all 2024-03-24T11:06:16.827 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-03-24T11:06:16.893 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-03-24T11:06:16.895 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-03-24T11:06:17.035 INFO:teuthology.orchestra.run.smithi100.stdout:74 files removed 2024-03-24T11:06:17.062 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-radosgw 2024-03-24T11:06:17.148 INFO:teuthology.orchestra.run.smithi105.stdout:74 files removed 2024-03-24T11:06:17.178 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2024-03-24T11:06:17.769 INFO:teuthology.orchestra.run.smithi100.stdout:ceph packages for x86_64 188 kB/s | 77 kB 00:00 2024-03-24T11:06:17.881 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 191 kB/s | 77 kB 00:00 2024-03-24T11:06:18.145 INFO:teuthology.orchestra.run.smithi100.stdout:ceph noarch packages 28 kB/s | 9.2 kB 00:00 2024-03-24T11:06:18.290 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 25 kB/s | 9.2 kB 00:00 2024-03-24T11:06:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:ceph source packages 5.0 kB/s | 1.8 kB 00:00 2024-03-24T11:06:18.632 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 5.9 kB/s | 1.8 kB 00:00 2024-03-24T11:06:18.956 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - AppStream 18 MB/s | 6.7 MB 00:00 2024-03-24T11:06:19.077 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - AppStream 16 MB/s | 6.7 MB 00:00 2024-03-24T11:06:20.423 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS-9 - BaseOS 5.7 MB/s | 2.0 MB 00:00 2024-03-24T11:06:20.460 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS-9 - BaseOS 7.3 MB/s | 2.0 MB 00:00 2024-03-24T11:06:22.743 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 4.1 MB/s | 8.0 MB 00:01 2024-03-24T11:06:29.438 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - BaseOS 946 kB/s | 8.0 MB 00:08 2024-03-24T11:06:31.546 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - AppStream 18 MB/s | 19 MB 00:01 2024-03-24T11:06:36.482 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - CRB 4.1 MB/s | 6.1 MB 00:01 2024-03-24T11:06:37.941 INFO:teuthology.orchestra.run.smithi100.stdout:CentOS Stream 9 - Extras packages 50 kB/s | 16 kB 00:00 2024-03-24T11:06:39.153 INFO:teuthology.orchestra.run.smithi100.stdout:Extra Packages for Enterprise Linux 19 MB/s | 21 MB 00:01 2024-03-24T11:06:44.034 INFO:teuthology.orchestra.run.smithi100.stdout:lab-extras 31 kB/s | 1.7 kB 00:00 2024-03-24T11:06:46.444 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 14 M 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrading: 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout: librados2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.1 M 2024-03-24T11:06:46.461 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 5.0 M 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 23 M 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 636 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 457 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 4.9 M 2024-03-24T11:06:46.462 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 45 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 117 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 156 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 320 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 293 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 99 k 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout:Install 22 Packages 2024-03-24T11:06:46.463 INFO:teuthology.orchestra.run.smithi100.stdout:Upgrade 2 Packages 2024-03-24T11:06:46.464 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:06:46.464 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 58 M 2024-03-24T11:06:46.464 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:06:47.793 INFO:teuthology.orchestra.run.smithi100.stdout:(1/24): ceph-base-17.2.7-726.g9c4e0c34.el9.x86_ 3.9 MB/s | 5.0 MB 00:01 2024-03-24T11:06:47.885 INFO:teuthology.orchestra.run.smithi100.stdout:(2/24): ceph-selinux-17.2.7-726.g9c4e0c34.el9.x 275 kB/s | 25 kB 00:00 2024-03-24T11:06:48.077 INFO:teuthology.orchestra.run.smithi100.stdout:(3/24): ceph-radosgw-17.2.7-726.g9c4e0c34.el9.x 9.0 MB/s | 14 MB 00:01 2024-03-24T11:06:48.119 INFO:teuthology.orchestra.run.smithi100.stdout:(4/24): libcephfs2-17.2.7-726.g9c4e0c34.el9.x86 2.7 MB/s | 636 kB 00:00 2024-03-24T11:06:48.144 INFO:teuthology.orchestra.run.smithi100.stdout:(5/24): libradosstriper1-17.2.7-726.g9c4e0c34.e 6.7 MB/s | 457 kB 00:00 2024-03-24T11:06:48.203 INFO:teuthology.orchestra.run.smithi100.stdout:(6/24): python3-ceph-argparse-17.2.7-726.g9c4e0 772 kB/s | 45 kB 00:00 2024-03-24T11:06:48.303 INFO:teuthology.orchestra.run.smithi100.stdout:(7/24): python3-ceph-common-17.2.7-726.g9c4e0c3 1.1 MB/s | 117 kB 00:00 2024-03-24T11:06:48.512 INFO:teuthology.orchestra.run.smithi100.stdout:(8/24): ceph-common-17.2.7-726.g9c4e0c34.el9.x8 11 MB/s | 23 MB 00:02 2024-03-24T11:06:48.537 INFO:teuthology.orchestra.run.smithi100.stdout:(9/24): python3-cephfs-17.2.7-726.g9c4e0c34.el9 670 kB/s | 156 kB 00:00 2024-03-24T11:06:48.562 INFO:teuthology.orchestra.run.smithi100.stdout:(10/24): python3-rados-17.2.7-726.g9c4e0c34.el9 6.3 MB/s | 320 kB 00:00 2024-03-24T11:06:48.613 INFO:teuthology.orchestra.run.smithi100.stdout:(11/24): python3-rbd-17.2.7-726.g9c4e0c34.el9.x 3.8 MB/s | 293 kB 00:00 2024-03-24T11:06:48.638 INFO:teuthology.orchestra.run.smithi100.stdout:(12/24): python3-rgw-17.2.7-726.g9c4e0c34.el9.x 1.3 MB/s | 99 kB 00:00 2024-03-24T11:06:48.772 INFO:teuthology.orchestra.run.smithi100.stdout:(13/24): librgw2-17.2.7-726.g9c4e0c34.el9.x86_6 7.5 MB/s | 4.9 MB 00:00 2024-03-24T11:06:48.814 INFO:teuthology.orchestra.run.smithi100.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 293 kB/s | 44 kB 00:00 2024-03-24T11:06:48.839 INFO:teuthology.orchestra.run.smithi100.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 704 kB/s | 159 kB 00:00 2024-03-24T11:06:48.948 INFO:teuthology.orchestra.run.smithi100.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 3.7 MB/s | 661 kB 00:00 2024-03-24T11:06:48.973 INFO:teuthology.orchestra.run.smithi100.stdout:(17/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 1.7 MB/s | 282 kB 00:00 2024-03-24T11:06:48.998 INFO:teuthology.orchestra.run.smithi100.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 259 kB/s | 41 kB 00:00 2024-03-24T11:06:49.090 INFO:teuthology.orchestra.run.smithi100.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 735 kB/s | 67 kB 00:00 2024-03-24T11:06:49.115 INFO:teuthology.orchestra.run.smithi100.stdout:(20/24): liboath-2.6.7-2.el9.x86_64.rpm 416 kB/s | 49 kB 00:00 2024-03-24T11:06:49.142 INFO:teuthology.orchestra.run.smithi100.stdout:(21/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-24T11:06:49.241 INFO:teuthology.orchestra.run.smithi100.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 11 MB/s | 1.6 MB 00:00 2024-03-24T11:06:49.391 INFO:teuthology.orchestra.run.smithi100.stdout:(23/24): librbd1-17.2.7-726.g9c4e0c34.el9.x86_6 12 MB/s | 2.9 MB 00:00 2024-03-24T11:06:49.491 INFO:teuthology.orchestra.run.smithi100.stdout:(24/24): librados2-17.2.7-726.g9c4e0c34.el9.x86 8.2 MB/s | 3.1 MB 00:00 2024-03-24T11:06:49.495 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:06:49.495 INFO:teuthology.orchestra.run.smithi100.stdout:Total 19 MB/s | 58 MB 00:03 2024-03-24T11:06:49.496 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:06:49.525 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:06:49.525 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:06:49.988 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:06:49.989 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:06:51.225 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:06:51.555 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-24T11:06:51.566 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 694 kB/s | 19 MB 00:27 2024-03-24T11:06:51.618 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-24T11:06:51.693 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-24T11:06:51.772 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-24T11:06:51.803 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-24T11:06:51.874 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-24T11:06:51.925 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-24T11:06:51.958 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-24T11:06:52.392 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-24T11:06:52.423 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-24T11:06:52.468 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-24T11:06:52.505 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-24T11:06:52.548 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 10/26 2024-03-24T11:06:52.585 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-24T11:06:52.647 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-24T11:06:52.675 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-24T11:06:52.741 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-24T11:06:52.780 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-24T11:06:52.816 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-24T11:06:52.857 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-24T11:06:53.159 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-24T11:06:53.187 INFO:teuthology.orchestra.run.smithi100.stdout: Upgrading : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-24T11:06:53.260 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-24T11:06:53.333 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 19/26 2024-03-24T11:06:53.357 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 20/26 2024-03-24T11:06:55.758 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-24T11:06:55.803 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-24T11:06:56.343 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-24T11:06:56.381 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-24T11:06:56.450 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 4.4 MB/s | 6.1 MB 00:01 2024-03-24T11:06:56.463 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-24T11:06:56.464 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:06:56.464 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:06:56.464 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:06:56.499 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-24T11:06:58.226 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 36 kB/s | 16 kB 00:00 2024-03-24T11:06:59.153 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 25 MB/s | 21 MB 00:00 2024-03-24T11:07:03.978 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 29 kB/s | 1.7 kB 00:00 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /sys 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /proc 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /mnt 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /var/tmp 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /home 2024-03-24T11:07:05.938 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /root 2024-03-24T11:07:05.939 INFO:teuthology.orchestra.run.smithi100.stdout:skipping the directory /tmp 2024-03-24T11:07:05.939 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:06.413 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:07:06.423 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:06.423 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:07:06.423 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:06.423 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 14 M 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.1 M 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 5.0 M 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 23 M 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 636 k 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-03-24T11:07:06.424 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 CentOS-AppStream 159 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 CentOS-AppStream 44 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 457 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 CentOS-AppStream 661 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 4.9 M 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 CentOS-AppStream 282 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 45 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 117 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 156 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 CentOS-AppStream 41 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 320 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 293 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 99 k 2024-03-24T11:07:06.425 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout:Install 22 Packages 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:06.426 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 58 M 2024-03-24T11:07:06.427 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:07:07.138 INFO:teuthology.orchestra.run.smithi105.stdout:(1/24): ceph-base-17.2.7-726.g9c4e0c34.el9.x86_ 7.5 MB/s | 5.0 MB 00:00 2024-03-24T11:07:07.198 INFO:teuthology.orchestra.run.smithi105.stdout:(2/24): ceph-selinux-17.2.7-726.g9c4e0c34.el9.x 433 kB/s | 25 kB 00:00 2024-03-24T11:07:07.288 INFO:teuthology.orchestra.run.smithi105.stdout:(3/24): libcephfs2-17.2.7-726.g9c4e0c34.el9.x86 6.8 MB/s | 636 kB 00:00 2024-03-24T11:07:07.336 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-24T11:07:07.380 INFO:teuthology.orchestra.run.smithi105.stdout:(4/24): libradosstriper1-17.2.7-726.g9c4e0c34.e 4.9 MB/s | 457 kB 00:00 2024-03-24T11:07:07.405 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-24T11:07:07.405 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:07.405 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:07:07.406 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:07:07.406 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:07:07.406 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:07.407 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-24T11:07:07.489 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-24T11:07:07.489 INFO:teuthology.orchestra.run.smithi100.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-24T11:07:07.947 INFO:teuthology.orchestra.run.smithi105.stdout:(5/24): librgw2-17.2.7-726.g9c4e0c34.el9.x86_64 8.7 MB/s | 4.9 MB 00:00 2024-03-24T11:07:07.997 INFO:teuthology.orchestra.run.smithi105.stdout:(6/24): python3-ceph-argparse-17.2.7-726.g9c4e0 906 kB/s | 45 kB 00:00 2024-03-24T11:07:08.214 INFO:teuthology.orchestra.run.smithi105.stdout:(7/24): ceph-common-17.2.7-726.g9c4e0c34.el9.x8 13 MB/s | 23 MB 00:01 2024-03-24T11:07:08.239 INFO:teuthology.orchestra.run.smithi105.stdout:(8/24): python3-ceph-common-17.2.7-726.g9c4e0c3 486 kB/s | 117 kB 00:00 2024-03-24T11:07:08.264 INFO:teuthology.orchestra.run.smithi105.stdout:(9/24): python3-cephfs-17.2.7-726.g9c4e0c34.el9 3.1 MB/s | 156 kB 00:00 2024-03-24T11:07:08.307 INFO:teuthology.orchestra.run.smithi105.stdout:(10/24): python3-rados-17.2.7-726.g9c4e0c34.el9 4.7 MB/s | 320 kB 00:00 2024-03-24T11:07:08.332 INFO:teuthology.orchestra.run.smithi105.stdout:(11/24): python3-rbd-17.2.7-726.g9c4e0c34.el9.x 4.3 MB/s | 293 kB 00:00 2024-03-24T11:07:08.357 INFO:teuthology.orchestra.run.smithi105.stdout:(12/24): python3-rgw-17.2.7-726.g9c4e0c34.el9.x 1.9 MB/s | 99 kB 00:00 2024-03-24T11:07:08.482 INFO:teuthology.orchestra.run.smithi105.stdout:(13/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 353 kB/s | 44 kB 00:00 2024-03-24T11:07:08.508 INFO:teuthology.orchestra.run.smithi105.stdout:(14/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 907 kB/s | 159 kB 00:00 2024-03-24T11:07:08.558 INFO:teuthology.orchestra.run.smithi105.stdout:(15/24): librdkafka-1.6.1-102.el9.x86_64.rpm 8.6 MB/s | 661 kB 00:00 2024-03-24T11:07:08.583 INFO:teuthology.orchestra.run.smithi105.stdout:(16/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 3.7 MB/s | 282 kB 00:00 2024-03-24T11:07:08.600 INFO:teuthology.orchestra.run.smithi105.stdout:(17/24): python3-prettytable-0.7.2-27.el9.noarc 990 kB/s | 41 kB 00:00 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 7/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 8/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 9/26 2024-03-24T11:07:08.659 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 10/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-24T11:07:08.660 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-24T11:07:08.661 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 25/26 2024-03-24T11:07:08.701 INFO:teuthology.orchestra.run.smithi105.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 486 kB/s | 49 kB 00:00 2024-03-24T11:07:08.767 INFO:teuthology.orchestra.run.smithi105.stdout:(19/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 1.6 MB/s | 309 kB 00:00 2024-03-24T11:07:08.793 INFO:teuthology.orchestra.run.smithi105.stdout:(20/24): libunwind-1.6.2-1.el9.x86_64.rpm 733 kB/s | 67 kB 00:00 2024-03-24T11:07:08.985 INFO:teuthology.orchestra.run.smithi105.stdout:(21/24): thrift-0.15.0-2.el9.x86_64.rpm 7.3 MB/s | 1.6 MB 00:00 2024-03-24T11:07:09.060 INFO:teuthology.orchestra.run.smithi105.stdout:(22/24): librados2-17.2.7-726.g9c4e0c34.el9.x86 12 MB/s | 3.1 MB 00:00 2024-03-24T11:07:09.246 INFO:teuthology.orchestra.run.smithi105.stdout:(23/24): librbd1-17.2.7-726.g9c4e0c34.el9.x86_6 11 MB/s | 2.9 MB 00:00 2024-03-24T11:07:09.631 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-24T11:07:09.631 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:09.631 INFO:teuthology.orchestra.run.smithi100.stdout:Upgraded: 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.632 INFO:teuthology.orchestra.run.smithi100.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.633 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.634 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:09.634 INFO:teuthology.orchestra.run.smithi100.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:07:09.634 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:09.634 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:10.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-test 2024-03-24T11:07:10.623 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:26 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:10.719 INFO:teuthology.orchestra.run.smithi105.stdout:(24/24): ceph-radosgw-17.2.7-726.g9c4e0c34.el9. 3.3 MB/s | 14 MB 00:04 2024-03-24T11:07:10.724 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:10.724 INFO:teuthology.orchestra.run.smithi105.stdout:Total 13 MB/s | 58 MB 00:04 2024-03-24T11:07:10.724 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:07:10.759 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:07:10.759 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:07:10.773 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 48 M 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 163 k 2024-03-24T11:07:10.774 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:10.775 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 49 M 2024-03-24T11:07:10.776 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 196 M 2024-03-24T11:07:10.776 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:11.051 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.4 MB/s | 217 kB 00:00 2024-03-24T11:07:11.101 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 5.9 MB/s | 300 kB 00:00 2024-03-24T11:07:11.135 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 1.8 MB/s | 63 kB 00:00 2024-03-24T11:07:11.177 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): libcephsqlite-17.2.7-726.g9c4e0c34.el9.x 577 kB/s | 163 kB 00:00 2024-03-24T11:07:11.221 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:07:11.221 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:07:11.961 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 227 kB/s | 187 kB 00:00 2024-03-24T11:07:12.436 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:07:12.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-03-24T11:07:12.813 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-24T11:07:12.904 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-24T11:07:12.983 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-24T11:07:13.006 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-24T11:07:13.077 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-24T11:07:13.139 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-03-24T11:07:13.171 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-03-24T11:07:13.511 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): ceph-test-17.2.7-726.g9c4e0c34.el9.x86_6 18 MB/s | 48 MB 00:02 2024-03-24T11:07:13.514 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:13.514 INFO:teuthology.orchestra.run.smithi100.stdout:Total 18 MB/s | 49 MB 00:02 2024-03-24T11:07:13.528 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:13.549 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:13.549 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:13.621 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-03-24T11:07:13.650 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-24T11:07:13.712 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 8/26 2024-03-24T11:07:13.750 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-03-24T11:07:13.792 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 10/26 2024-03-24T11:07:13.830 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-24T11:07:13.892 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-24T11:07:13.920 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-24T11:07:13.937 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:13.938 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:13.969 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-24T11:07:14.008 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-03-24T11:07:14.061 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-03-24T11:07:14.093 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-03-24T11:07:14.396 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-03-24T11:07:14.432 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-24T11:07:14.514 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:14.520 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 18/26 2024-03-24T11:07:14.586 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:07:14.594 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 19/26 2024-03-24T11:07:14.625 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 20/26 2024-03-24T11:07:14.628 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:07:14.660 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:07:14.710 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:07:14.739 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-24T11:07:16.968 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-24T11:07:17.006 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-24T11:07:17.550 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 21/26 2024-03-24T11:07:17.592 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-24T11:07:17.666 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 22/26 2024-03-24T11:07:17.666 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-03-24T11:07:17.666 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-03-24T11:07:17.667 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:17.719 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-24T11:07:19.189 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-24T11:07:19.280 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-24T11:07:21.590 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-24T11:07:21.590 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/6 2024-03-24T11:07:21.590 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/6 2024-03-24T11:07:21.591 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:07:21.591 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:07:21.591 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:07:21.977 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:07:21.977 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:21.977 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:21.978 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:22.256 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph 2024-03-24T11:07:22.818 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:38 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:22.970 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: ceph x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 6.5 k 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.1 M 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 1.4 M 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 240 k 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.7 M 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 15 M 2024-03-24T11:07:22.973 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:07:22.974 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:22.975 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:22.976 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================== 2024-03-24T11:07:22.976 INFO:teuthology.orchestra.run.smithi100.stdout:Install 33 Packages 2024-03-24T11:07:22.976 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:22.976 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 27 M 2024-03-24T11:07:22.976 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 100 M 2024-03-24T11:07:22.976 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:23.455 INFO:teuthology.orchestra.run.smithi100.stdout:(1/33): ceph-17.2.7-726.g9c4e0c34.el9.x86_64.rp 28 kB/s | 6.5 kB 00:00 2024-03-24T11:07:23.596 INFO:teuthology.orchestra.run.smithi100.stdout:(2/33): ceph-mds-17.2.7-726.g9c4e0c34.el9.x86_6 5.5 MB/s | 2.1 MB 00:00 2024-03-24T11:07:23.880 INFO:teuthology.orchestra.run.smithi100.stdout:(3/33): ceph-mgr-17.2.7-726.g9c4e0c34.el9.x86_6 2.1 MB/s | 1.4 MB 00:00 2024-03-24T11:07:23.997 INFO:teuthology.orchestra.run.smithi100.stdout:(4/33): ceph-mgr-modules-core-17.2.7-726.g9c4e0 2.0 MB/s | 240 kB 00:00 2024-03-24T11:07:24.122 INFO:teuthology.orchestra.run.smithi100.stdout:(5/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.1 MB/s | 144 kB 00:00 2024-03-24T11:07:24.156 INFO:teuthology.orchestra.run.smithi100.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 948 kB/s | 32 kB 00:00 2024-03-24T11:07:24.190 INFO:teuthology.orchestra.run.smithi100.stdout:(7/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 47 kB 00:00 2024-03-24T11:07:24.225 INFO:teuthology.orchestra.run.smithi100.stdout:(8/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T11:07:24.376 INFO:teuthology.orchestra.run.smithi100.stdout:(9/33): ceph-osd-17.2.7-726.g9c4e0c34.el9.x86_6 20 MB/s | 15 MB 00:00 2024-03-24T11:07:24.401 INFO:teuthology.orchestra.run.smithi100.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 1.3 MB/s | 241 kB 00:00 2024-03-24T11:07:24.460 INFO:teuthology.orchestra.run.smithi100.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 14 MB/s | 1.1 MB 00:00 2024-03-24T11:07:24.485 INFO:teuthology.orchestra.run.smithi100.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.2 MB/s | 103 kB 00:00 2024-03-24T11:07:24.510 INFO:teuthology.orchestra.run.smithi100.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 2.4 MB/s | 124 kB 00:00 2024-03-24T11:07:24.544 INFO:teuthology.orchestra.run.smithi100.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 3.1 MB/s | 187 kB 00:00 2024-03-24T11:07:24.644 INFO:teuthology.orchestra.run.smithi100.stdout:(15/33): ceph-mon-17.2.7-726.g9c4e0c34.el9.x86_ 3.1 MB/s | 3.7 MB 00:01 2024-03-24T11:07:24.753 INFO:teuthology.orchestra.run.smithi100.stdout:(16/33): python3-requests-2.25.1-8.el9.noarch.r 516 kB/s | 125 kB 00:00 2024-03-24T11:07:25.095 INFO:teuthology.orchestra.run.smithi100.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 79 kB/s | 43 kB 00:00 2024-03-24T11:07:25.195 INFO:teuthology.orchestra.run.smithi100.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 107 kB/s | 11 kB 00:00 2024-03-24T11:07:25.303 INFO:teuthology.orchestra.run.smithi100.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 163 kB/s | 18 kB 00:00 2024-03-24T11:07:25.329 INFO:teuthology.orchestra.run.smithi100.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 260 kB/s | 172 kB 00:00 2024-03-24T11:07:25.412 INFO:teuthology.orchestra.run.smithi100.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 213 kB/s | 23 kB 00:00 2024-03-24T11:07:25.438 INFO:teuthology.orchestra.run.smithi100.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 179 kB/s | 19 kB 00:00 2024-03-24T11:07:25.513 INFO:teuthology.orchestra.run.smithi100.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 196 kB/s | 20 kB 00:00 2024-03-24T11:07:25.538 INFO:teuthology.orchestra.run.smithi100.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 461 kB/s | 46 kB 00:00 2024-03-24T11:07:25.697 INFO:teuthology.orchestra.run.smithi100.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 429 kB/s | 79 kB 00:00 2024-03-24T11:07:25.722 INFO:teuthology.orchestra.run.smithi100.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 379 kB/s | 358 kB 00:00 2024-03-24T11:07:25.806 INFO:teuthology.orchestra.run.smithi100.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-24T11:07:25.906 INFO:teuthology.orchestra.run.smithi100.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 489 kB/s | 90 kB 00:00 2024-03-24T11:07:25.931 INFO:teuthology.orchestra.run.smithi100.stdout:(29/33): python3-pecan-1.4.2-3.el9.noarch.rpm 693 kB/s | 272 kB 00:00 2024-03-24T11:07:25.956 INFO:teuthology.orchestra.run.smithi100.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 238 kB/s | 36 kB 00:00 2024-03-24T11:07:26.069 INFO:teuthology.orchestra.run.smithi100.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 179 kB/s | 20 kB 00:00 2024-03-24T11:07:26.273 INFO:teuthology.orchestra.run.smithi100.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 625 kB/s | 230 kB 00:00 2024-03-24T11:07:26.574 INFO:teuthology.orchestra.run.smithi100.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 692 kB/s | 427 kB 00:00 2024-03-24T11:07:26.580 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:26.580 INFO:teuthology.orchestra.run.smithi100.stdout:Total 7.4 MB/s | 27 MB 00:03 2024-03-24T11:07:26.594 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:26.635 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:26.635 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:26.965 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:26.965 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:27.355 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-24T11:07:27.355 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-03-24T11:07:27.355 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-03-24T11:07:27.356 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-03-24T11:07:27.356 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-03-24T11:07:27.356 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-03-24T11:07:27.356 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-03-24T11:07:27.356 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-03-24T11:07:27.356 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:28.263 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:28.327 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-24T11:07:28.367 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-24T11:07:28.400 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-24T11:07:28.492 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-24T11:07:28.578 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-24T11:07:28.619 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-24T11:07:28.632 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-24T11:07:28.691 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-24T11:07:28.693 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 24/26 2024-03-24T11:07:28.694 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:28.694 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-03-24T11:07:28.694 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:07:28.694 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-03-24T11:07:28.694 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:28.695 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-24T11:07:28.745 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-24T11:07:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-03-24T11:07:28.760 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-24T11:07:28.783 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-24T11:07:28.841 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-24T11:07:28.913 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-24T11:07:29.097 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-24T11:07:29.146 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-24T11:07:29.177 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-24T11:07:29.219 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-24T11:07:29.258 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-24T11:07:29.333 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-24T11:07:29.368 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T11:07:29.401 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-24T11:07:29.442 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-24T11:07:29.477 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-24T11:07:29.512 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-24T11:07:29.600 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-24T11:07:29.642 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-24T11:07:29.722 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-24T11:07:29.867 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-24T11:07:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-03-24T11:07:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/26 2024-03-24T11:07:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/26 2024-03-24T11:07:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/26 2024-03-24T11:07:29.971 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 7/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x 8/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86 9/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 10/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 11/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 12/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 13/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-03-24T11:07:29.972 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 23/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-03-24T11:07:29.973 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 25/26 2024-03-24T11:07:30.000 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T11:07:30.149 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 28/33 2024-03-24T11:07:30.172 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-24T11:07:30.196 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-24T11:07:30.196 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:30.196 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:07:30.197 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:07:30.197 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:07:30.197 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:30.775 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-03-24T11:07:30.775 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:30.775 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-03-24T11:07:30.775 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.775 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.775 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-03-24T11:07:30.776 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:30.777 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:07:31.145 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2024-03-24T11:07:31.540 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-24T11:07:31.566 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-24T11:07:31.566 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:31.566 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:07:31.566 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:07:31.566 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:07:31.566 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:31.694 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:28 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:07:31.841 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:07:31.841 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:31.841 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:07:31.841 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 48 M 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: jq x86_64 1.6-16.el9 baseos 187 k 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 163 k 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma x86_64 6.9.6-1.el9.5 CentOS-AppStream 217 k 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-5.el9 CentOS-AppStream 300 k 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 CentOS-AppStream 63 k 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-03-24T11:07:31.842 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:31.843 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 49 M 2024-03-24T11:07:31.843 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 196 M 2024-03-24T11:07:31.843 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:07:31.958 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-24T11:07:31.982 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-24T11:07:31.982 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:31.983 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:07:31.983 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:07:31.983 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:07:31.983 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:32.098 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): oniguruma-6.9.6-1.el9.5.x86_64.rpm 1.2 MB/s | 217 kB 00:00 2024-03-24T11:07:32.140 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): socat-1.7.4.1-5.el9.x86_64.rpm 7.0 MB/s | 300 kB 00:00 2024-03-24T11:07:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): xmlstarlet-1.6.1-20.el9.x86_64.rpm 2.5 MB/s | 63 kB 00:00 2024-03-24T11:07:32.192 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-24T11:07:32.219 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-24T11:07:32.220 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:32.220 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:07:32.220 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:07:32.220 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:07:32.220 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:32.282 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-24T11:07:32.315 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): libcephsqlite-17.2.7-726.g9c4e0c34.el9.x 418 kB/s | 163 kB 00:00 2024-03-24T11:07:32.407 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): jq-1.6-16.el9.x86_64.rpm 773 kB/s | 187 kB 00:00 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 6/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-24T11:07:33.982 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-24T11:07:33.983 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-24T11:07:33.984 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-24T11:07:33.984 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-24T11:07:33.984 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-24T11:07:33.985 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T11:07:33.985 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-24T11:07:33.985 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-24T11:07:33.985 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-24T11:07:33.985 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-24T11:07:33.985 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-24T11:07:34.466 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): ceph-test-17.2.7-726.g9c4e0c34.el9.x86_6 19 MB/s | 48 MB 00:02 2024-03-24T11:07:34.470 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:34.470 INFO:teuthology.orchestra.run.smithi105.stdout:Total 19 MB/s | 49 MB 00:02 2024-03-24T11:07:34.485 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:07:34.505 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:07:34.506 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:07:34.876 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-24T11:07:34.876 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:07:34.877 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:07:34.878 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:07:34.878 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:07:34.878 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:07:34.878 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:07:34.878 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:07:34.878 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:07:34.879 INFO:teuthology.orchestra.run.smithi100.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:07:34.880 INFO:teuthology.orchestra.run.smithi100.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:07:34.881 INFO:teuthology.orchestra.run.smithi100.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:07:34.881 INFO:teuthology.orchestra.run.smithi100.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:07:34.881 INFO:teuthology.orchestra.run.smithi100.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:07:34.881 INFO:teuthology.orchestra.run.smithi100.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:07:34.881 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:34.881 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:34.891 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:07:34.892 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:07:35.265 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-base 2024-03-24T11:07:35.470 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:07:35.550 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/6 2024-03-24T11:07:35.583 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-5.el9.x86_64 2/6 2024-03-24T11:07:35.615 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:07:35.658 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : jq-1.6-16.el9.x86_64 4/6 2024-03-24T11:07:35.677 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-24T11:07:35.866 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:51 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:35.962 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:07:36.013 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:36.014 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:07:36.015 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:36.073 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install cephadm 2024-03-24T11:07:36.629 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:52 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:36.775 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:36.776 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:36.776 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:07:36.776 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:36.776 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:36.776 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 82 k 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 82 k 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 372 k 2024-03-24T11:07:36.777 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:36.984 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm-17.2.7-726.g9c4e0c34.el9.noarch.rpm 397 kB/s | 82 kB 00:00 2024-03-24T11:07:36.984 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:36.984 INFO:teuthology.orchestra.run.smithi100.stdout:Total 394 kB/s | 82 kB 00:00 2024-03-24T11:07:36.985 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:36.990 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:36.990 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:36.991 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:36.992 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:37.063 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:37.405 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:37.497 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:37.902 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:38.179 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:38.179 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:38.179 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:38.398 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:07:38.968 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:39.118 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:39.119 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:39.119 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-24T11:07:39.119 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 137 k 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 137 k 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 379 k 2024-03-24T11:07:39.120 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:39.455 INFO:teuthology.orchestra.run.smithi100.stdout:ceph-immutable-object-cache-17.2.7-726.g9c4e0c3 412 kB/s | 137 kB 00:00 2024-03-24T11:07:39.455 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:39.455 INFO:teuthology.orchestra.run.smithi100.stdout:Total 410 kB/s | 137 kB 00:00 2024-03-24T11:07:39.455 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:39.462 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:39.462 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:39.514 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:39.514 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:39.653 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:39.727 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-24T11:07:39.751 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-24T11:07:39.751 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:39.752 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:07:39.752 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:40.122 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/6 2024-03-24T11:07:40.193 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-24T11:07:40.699 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-24T11:07:40.700 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:40.700 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:40.700 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:40.700 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:40.700 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:40.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr 2024-03-24T11:07:41.522 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:57 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:41.619 INFO:teuthology.orchestra.run.smithi100.stdout:Package ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:07:41.671 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:41.672 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:07:41.672 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:41.729 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:07:42.281 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:00:58 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:42.431 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:42.432 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:42.432 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:07:42.432 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:42.432 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:42.432 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 1.7 M 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 23 k 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 15 k 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout:Installing weak dependencies: 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout:Install 7 Packages 2024-03-24T11:07:42.433 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:42.434 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.0 M 2024-03-24T11:07:42.434 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 8.1 M 2024-03-24T11:07:42.434 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:42.477 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 6/6 2024-03-24T11:07:42.477 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/6 2024-03-24T11:07:42.477 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/6 2024-03-24T11:07:42.477 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : oniguruma-6.9.6-1.el9.5.x86_64 3/6 2024-03-24T11:07:42.477 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-5.el9.x86_64 4/6 2024-03-24T11:07:42.477 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 5/6 2024-03-24T11:07:42.857 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : jq-1.6-16.el9.x86_64 6/6 2024-03-24T11:07:42.857 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:42.857 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: jq-1.6-16.el9.x86_64 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: oniguruma-6.9.6-1.el9.5.x86_64 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-5.el9.x86_64 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:42.858 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:07:42.964 INFO:teuthology.orchestra.run.smithi100.stdout:(1/7): ceph-grafana-dashboards-17.2.7-726.g9c4e 125 kB/s | 23 kB 00:00 2024-03-24T11:07:43.056 INFO:teuthology.orchestra.run.smithi100.stdout:(2/7): ceph-prometheus-alerts-17.2.7-726.g9c4e0 52 kB/s | 15 kB 00:00 2024-03-24T11:07:43.139 INFO:teuthology.orchestra.run.smithi100.stdout:(3/7): ceph-mgr-dashboard-17.2.7-726.g9c4e0c34. 4.7 MB/s | 1.7 MB 00:00 2024-03-24T11:07:43.160 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2024-03-24T11:07:43.290 INFO:teuthology.orchestra.run.smithi100.stdout:(4/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 28 kB/s | 9.0 kB 00:00 2024-03-24T11:07:43.465 INFO:teuthology.orchestra.run.smithi100.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 106 kB/s | 41 kB 00:00 2024-03-24T11:07:43.506 INFO:teuthology.orchestra.run.smithi100.stdout:(6/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 84 kB/s | 31 kB 00:00 2024-03-24T11:07:43.658 INFO:teuthology.orchestra.run.smithi100.stdout:(7/7): python3-routes-2.5.1-5.el9.noarch.rpm 514 kB/s | 188 kB 00:00 2024-03-24T11:07:43.660 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:43.661 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.6 MB/s | 2.0 MB 00:01 2024-03-24T11:07:43.661 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:43.668 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:43.668 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:43.726 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:40 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:07:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:43.728 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:43.879 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:07:43.881 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-24T11:07:43.882 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:07:43.882 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-24T11:07:43.882 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:07:43.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 6.5 k 2024-03-24T11:07:43.882 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:07:43.882 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.1 M 2024-03-24T11:07:43.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 1.4 M 2024-03-24T11:07:43.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 240 k 2024-03-24T11:07:43.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 3.7 M 2024-03-24T11:07:43.883 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 15 M 2024-03-24T11:07:43.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-03-24T11:07:43.883 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 CentOS-BaseOS 241 k 2024-03-24T11:07:43.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-03-24T11:07:43.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-03-24T11:07:43.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 CentOS-BaseOS 1.1 M 2024-03-24T11:07:43.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-03-24T11:07:43.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-03-24T11:07:43.884 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections noarch 3.0.0-8.el9 epel 23 k 2024-03-24T11:07:43.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-03-24T11:07:43.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-03-24T11:07:43.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-03-24T11:07:43.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 CentOS-AppStream 144 k 2024-03-24T11:07:43.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 CentOS-AppStream 32 k 2024-03-24T11:07:43.885 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-03-24T11:07:43.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-03-24T11:07:43.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 CentOS-BaseOS 103 k 2024-03-24T11:07:43.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-03-24T11:07:43.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-03-24T11:07:43.886 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 CentOS-BaseOS 124 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 CentOS-AppStream 47 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 CentOS-AppStream 39 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 CentOS-BaseOS 187 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-03-24T11:07:43.887 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-03-24T11:07:43.888 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-03-24T11:07:43.888 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:43.888 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:07:43.888 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================== 2024-03-24T11:07:43.888 INFO:teuthology.orchestra.run.smithi105.stdout:Install 33 Packages 2024-03-24T11:07:43.888 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:43.889 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 27 M 2024-03-24T11:07:43.890 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 100 M 2024-03-24T11:07:43.891 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:07:44.020 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:44.083 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-24T11:07:44.117 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-24T11:07:44.163 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-24T11:07:44.204 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-24T11:07:44.232 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 5/7 2024-03-24T11:07:44.458 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 6/7 2024-03-24T11:07:44.502 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-24T11:07:44.545 INFO:teuthology.orchestra.run.smithi105.stdout:(1/33): ceph-17.2.7-726.g9c4e0c34.el9.x86_64.rp 32 kB/s | 6.5 kB 00:00 2024-03-24T11:07:44.922 INFO:teuthology.orchestra.run.smithi105.stdout:(2/33): ceph-mds-17.2.7-726.g9c4e0c34.el9.x86_6 3.6 MB/s | 2.1 MB 00:00 2024-03-24T11:07:44.962 INFO:teuthology.orchestra.run.smithi105.stdout:(3/33): ceph-mgr-17.2.7-726.g9c4e0c34.el9.x86_6 2.3 MB/s | 1.4 MB 00:00 2024-03-24T11:07:45.001 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-24T11:07:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 1/7 2024-03-24T11:07:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2/7 2024-03-24T11:07:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 3/7 2024-03-24T11:07:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-24T11:07:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-24T11:07:45.002 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-24T11:07:45.063 INFO:teuthology.orchestra.run.smithi105.stdout:(4/33): ceph-mgr-modules-core-17.2.7-726.g9c4e0 2.3 MB/s | 240 kB 00:00 2024-03-24T11:07:45.155 INFO:teuthology.orchestra.run.smithi105.stdout:(5/33): ceph-mon-17.2.7-726.g9c4e0c34.el9.x86_6 6.1 MB/s | 3.7 MB 00:00 2024-03-24T11:07:45.188 INFO:teuthology.orchestra.run.smithi105.stdout:(6/33): python3-markupsafe-1.1.1-12.el9.x86_64. 984 kB/s | 32 kB 00:00 2024-03-24T11:07:45.213 INFO:teuthology.orchestra.run.smithi105.stdout:(7/33): python3-mako-1.1.4-6.el9.noarch.rpm 958 kB/s | 144 kB 00:00 2024-03-24T11:07:45.231 INFO:teuthology.orchestra.run.smithi105.stdout:(8/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.1 MB/s | 47 kB 00:00 2024-03-24T11:07:45.247 INFO:teuthology.orchestra.run.smithi105.stdout:(9/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.1 MB/s | 39 kB 00:00 2024-03-24T11:07:45.273 INFO:teuthology.orchestra.run.smithi105.stdout:(10/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 5.6 MB/s | 241 kB 00:00 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:45.302 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-24T11:07:45.303 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-24T11:07:45.303 INFO:teuthology.orchestra.run.smithi100.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:07:45.303 INFO:teuthology.orchestra.run.smithi100.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:07:45.303 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:45.303 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:45.348 INFO:teuthology.orchestra.run.smithi105.stdout:(11/33): python3-cryptography-36.0.1-4.el9.x86_ 11 MB/s | 1.1 MB 00:00 2024-03-24T11:07:45.373 INFO:teuthology.orchestra.run.smithi105.stdout:(12/33): python3-ply-3.11-14.el9.noarch.rpm 1.0 MB/s | 103 kB 00:00 2024-03-24T11:07:45.428 INFO:teuthology.orchestra.run.smithi105.stdout:(13/33): python3-pycparser-2.20-6.el9.noarch.rp 1.5 MB/s | 124 kB 00:00 2024-03-24T11:07:45.457 INFO:teuthology.orchestra.run.smithi105.stdout:(14/33): python3-urllib3-1.26.5-5.el9.noarch.rp 2.2 MB/s | 187 kB 00:00 2024-03-24T11:07:45.571 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:07:45.641 INFO:teuthology.orchestra.run.smithi105.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 587 kB/s | 125 kB 00:00 2024-03-24T11:07:46.000 INFO:teuthology.orchestra.run.smithi105.stdout:(16/33): ceph-osd-17.2.7-726.g9c4e0c34.el9.x86_ 14 MB/s | 15 MB 00:01 2024-03-24T11:07:46.025 INFO:teuthology.orchestra.run.smithi105.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 77 kB/s | 43 kB 00:00 2024-03-24T11:07:46.126 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:02 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:46.133 INFO:teuthology.orchestra.run.smithi105.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 98 kB/s | 11 kB 00:00 2024-03-24T11:07:46.250 INFO:teuthology.orchestra.run.smithi105.stdout:(19/33): python3-jaraco-classes-3.2.1-5.el9.noa 151 kB/s | 18 kB 00:00 2024-03-24T11:07:46.276 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:46.277 INFO:teuthology.orchestra.run.smithi100.stdout:=========================================================================================== 2024-03-24T11:07:46.277 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:07:46.277 INFO:teuthology.orchestra.run.smithi100.stdout:=========================================================================================== 2024-03-24T11:07:46.277 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:46.277 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 7.4 M 2024-03-24T11:07:46.277 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:07:46.278 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:46.279 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:46.279 INFO:teuthology.orchestra.run.smithi100.stdout:=========================================================================================== 2024-03-24T11:07:46.279 INFO:teuthology.orchestra.run.smithi100.stdout:Install 12 Packages 2024-03-24T11:07:46.279 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:46.279 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 37 M 2024-03-24T11:07:46.279 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 209 M 2024-03-24T11:07:46.280 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:46.367 INFO:teuthology.orchestra.run.smithi105.stdout:(20/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 237 kB/s | 172 kB 00:00 2024-03-24T11:07:46.393 INFO:teuthology.orchestra.run.smithi105.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 163 kB/s | 23 kB 00:00 2024-03-24T11:07:46.501 INFO:teuthology.orchestra.run.smithi105.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 145 kB/s | 19 kB 00:00 2024-03-24T11:07:46.526 INFO:teuthology.orchestra.run.smithi105.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 147 kB/s | 20 kB 00:00 2024-03-24T11:07:46.668 INFO:teuthology.orchestra.run.smithi105.stdout:(24/33): python3-more-itertools-8.12.0-2.el9.no 556 kB/s | 79 kB 00:00 2024-03-24T11:07:46.689 INFO:teuthology.orchestra.run.smithi100.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 210 kB/s | 32 kB 00:00 2024-03-24T11:07:46.694 INFO:teuthology.orchestra.run.smithi105.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 241 kB/s | 46 kB 00:00 2024-03-24T11:07:46.719 INFO:teuthology.orchestra.run.smithi105.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 499 kB/s | 358 kB 00:00 2024-03-24T11:07:46.722 INFO:teuthology.orchestra.run.smithi100.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 534 kB/s | 17 kB 00:00 2024-03-24T11:07:46.765 INFO:teuthology.orchestra.run.smithi100.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 849 kB/s | 34 kB 00:00 2024-03-24T11:07:46.803 INFO:teuthology.orchestra.run.smithi105.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 151 kB/s | 16 kB 00:00 2024-03-24T11:07:46.815 INFO:teuthology.orchestra.run.smithi100.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T11:07:46.836 INFO:teuthology.orchestra.run.smithi105.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 770 kB/s | 90 kB 00:00 2024-03-24T11:07:46.911 INFO:teuthology.orchestra.run.smithi105.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 330 kB/s | 36 kB 00:00 2024-03-24T11:07:46.970 INFO:teuthology.orchestra.run.smithi105.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 903 kB/s | 272 kB 00:00 2024-03-24T11:07:46.995 INFO:teuthology.orchestra.run.smithi105.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.4 MB/s | 230 kB 00:00 2024-03-24T11:07:46.999 INFO:teuthology.orchestra.run.smithi100.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-24T11:07:47.057 INFO:teuthology.orchestra.run.smithi100.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 20 MB/s | 4.8 MB 00:00 2024-03-24T11:07:47.062 INFO:teuthology.orchestra.run.smithi105.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 218 kB/s | 20 kB 00:00 2024-03-24T11:07:47.082 INFO:teuthology.orchestra.run.smithi100.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.6 MB/s | 217 kB 00:00 2024-03-24T11:07:47.145 INFO:teuthology.orchestra.run.smithi105.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 1.8 MB/s | 427 kB 00:00 2024-03-24T11:07:47.152 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:47.152 INFO:teuthology.orchestra.run.smithi105.stdout:Total 8.2 MB/s | 27 MB 00:03 2024-03-24T11:07:47.166 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:07:47.206 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:07:47.206 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:07:47.375 INFO:teuthology.orchestra.run.smithi100.stdout:(8/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 49 MB/s | 16 MB 00:00 2024-03-24T11:07:47.458 INFO:teuthology.orchestra.run.smithi100.stdout:(9/12): ceph-mgr-diskprediction-local-17.2.7-72 8.1 MB/s | 7.4 MB 00:00 2024-03-24T11:07:47.545 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:07:47.545 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:07:47.617 INFO:teuthology.orchestra.run.smithi100.stdout:(10/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.5 MB/s | 245 kB 00:00 2024-03-24T11:07:47.884 INFO:teuthology.orchestra.run.smithi100.stdout:(11/12): libquadmath-11.4.1-3.el9.x86_64.rpm 384 kB/s | 195 kB 00:00 2024-03-24T11:07:47.909 INFO:teuthology.orchestra.run.smithi100.stdout:(12/12): libgfortran-11.4.1-3.el9.x86_64.rpm 975 kB/s | 803 kB 00:00 2024-03-24T11:07:47.911 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:47.911 INFO:teuthology.orchestra.run.smithi100.stdout:Total 23 MB/s | 37 MB 00:01 2024-03-24T11:07:47.954 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:47.980 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:47.980 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:48.410 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:48.410 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:48.794 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:07:48.849 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-03-24T11:07:48.889 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-03-24T11:07:48.922 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-03-24T11:07:49.006 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-03-24T11:07:49.083 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-03-24T11:07:49.108 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:49.125 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-03-24T11:07:49.196 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:07:49.204 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-03-24T11:07:49.230 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:07:49.259 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 8/33 2024-03-24T11:07:49.297 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 9/33 2024-03-24T11:07:49.322 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:07:49.346 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 10/33 2024-03-24T11:07:49.366 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:07:49.404 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 11/33 2024-03-24T11:07:49.553 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:07:49.585 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 12/33 2024-03-24T11:07:49.590 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:07:49.651 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 13/33 2024-03-24T11:07:49.667 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:07:49.687 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 14/33 2024-03-24T11:07:49.725 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 15/33 2024-03-24T11:07:49.755 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 16/33 2024-03-24T11:07:49.822 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 17/33 2024-03-24T11:07:49.857 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T11:07:49.890 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 19/33 2024-03-24T11:07:49.931 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 20/33 2024-03-24T11:07:49.965 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 21/33 2024-03-24T11:07:50.000 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 22/33 2024-03-24T11:07:50.124 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 23/33 2024-03-24T11:07:50.164 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 24/33 2024-03-24T11:07:50.236 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 25/33 2024-03-24T11:07:50.321 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:07:50.374 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 26/33 2024-03-24T11:07:50.428 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:07:50.506 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T11:07:50.655 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 28/33 2024-03-24T11:07:50.686 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-24T11:07:50.709 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 29/33 2024-03-24T11:07:50.710 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:50.710 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-03-24T11:07:50.710 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:07:50.710 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-03-24T11:07:50.710 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:51.664 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:07:52.045 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-24T11:07:52.071 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 30/33 2024-03-24T11:07:52.071 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:52.071 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-03-24T11:07:52.072 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:07:52.072 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-03-24T11:07:52.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:52.454 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-24T11:07:52.477 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 31/33 2024-03-24T11:07:52.478 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:52.478 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-03-24T11:07:52.478 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:07:52.478 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-03-24T11:07:52.478 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:52.680 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-24T11:07:52.689 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:07:52.703 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 32/33 2024-03-24T11:07:52.703 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:07:52.703 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-03-24T11:07:52.703 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:07:52.704 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-03-24T11:07:52.704 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:52.724 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-24T11:07:52.761 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 33/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 3/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 4/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 5/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.n 6/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 7/33 2024-03-24T11:07:54.417 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 8/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 9/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 10/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 11/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 12/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 13/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 14/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 15/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 16/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-03-24T11:07:54.418 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-03-24T11:07:54.419 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-03-24T11:07:54.420 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-03-24T11:07:54.420 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-03-24T11:07:55.473 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-03-24T11:07:55.473 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:55.473 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:07:55.473 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-03-24T11:07:55.474 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-03-24T11:07:55.475 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:55.476 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 1/12 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:07:55.531 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:07:55.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:07:55.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:07:55.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:07:55.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:07:55.532 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:07:55.812 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:07:55.922 INFO:teuthology.orchestra.run.smithi100.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:55.923 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:07:56.309 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:07:56.378 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:53 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:07:56.474 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:07:56.524 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:07:56.525 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:07:56.525 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:07:56.594 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2024-03-24T11:07:56.873 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:12 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:07:57.023 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:07:57.024 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-03-24T11:07:57.024 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:07:57.024 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-03-24T11:07:57.024 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:07:57.024 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 47 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:07:57.025 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout:Install 13 Packages 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:07:57.026 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.0 M 2024-03-24T11:07:57.027 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 26 M 2024-03-24T11:07:57.027 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:07:57.157 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:54 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:07:57.313 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 82 k 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:57.314 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:07:57.315 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:57.315 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 82 k 2024-03-24T11:07:57.315 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 372 k 2024-03-24T11:07:57.315 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:07:57.499 INFO:teuthology.orchestra.run.smithi100.stdout:(1/13): python3-jsonpointer-2.0-4.el9.noarch.rp 135 kB/s | 18 kB 00:00 2024-03-24T11:07:57.525 INFO:teuthology.orchestra.run.smithi100.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 153 kB/s | 24 kB 00:00 2024-03-24T11:07:57.583 INFO:teuthology.orchestra.run.smithi100.stdout:(3/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-24T11:07:57.609 INFO:teuthology.orchestra.run.smithi100.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.5 MB/s | 132 kB 00:00 2024-03-24T11:07:57.634 INFO:teuthology.orchestra.run.smithi100.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T11:07:57.637 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-17.2.7-726.g9c4e0c34.el9.noarch.rpm 254 kB/s | 82 kB 00:00 2024-03-24T11:07:57.639 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:57.639 INFO:teuthology.orchestra.run.smithi105.stdout:Total 253 kB/s | 82 kB 00:00 2024-03-24T11:07:57.639 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:07:57.644 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:07:57.644 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:07:57.646 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:07:57.647 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:07:57.668 INFO:teuthology.orchestra.run.smithi100.stdout:(6/13): ceph-mgr-rook-17.2.7-726.g9c4e0c34.el9. 157 kB/s | 47 kB 00:00 2024-03-24T11:07:57.693 INFO:teuthology.orchestra.run.smithi100.stdout:(7/13): python3-requests-oauthlib-1.3.0-12.el9. 510 kB/s | 43 kB 00:00 2024-03-24T11:07:57.724 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:07:57.935 INFO:teuthology.orchestra.run.smithi100.stdout:(8/13): python3-certifi-2021.10.8-2.el9.noarch. 62 kB/s | 15 kB 00:00 2024-03-24T11:07:57.969 INFO:teuthology.orchestra.run.smithi100.stdout:(9/13): python3-cachetools-4.2.4-1.el9.noarch.r 96 kB/s | 32 kB 00:00 2024-03-24T11:07:58.184 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:58.185 INFO:teuthology.orchestra.run.smithi100.stdout:(10/13): python3-google-auth-2.28.2-1.el9.noarc 412 kB/s | 203 kB 00:00 2024-03-24T11:07:58.219 INFO:teuthology.orchestra.run.smithi100.stdout:(11/13): python3-rsa-4.9-2.el9.noarch.rpm 236 kB/s | 59 kB 00:00 2024-03-24T11:07:58.292 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:58.302 INFO:teuthology.orchestra.run.smithi100.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 768 kB/s | 90 kB 00:00 2024-03-24T11:07:58.494 INFO:teuthology.orchestra.run.smithi100.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 1.8 MB/s | 1.0 MB 00:00 2024-03-24T11:07:58.497 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:07:58.497 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.4 MB/s | 2.0 MB 00:01 2024-03-24T11:07:58.498 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:07:58.512 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:07:58.512 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:07:58.606 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:07:58.607 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:07:58.684 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:58.937 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/1 2024-03-24T11:07:58.938 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:58.938 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:07:58.938 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:07:58.938 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:58.938 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:07:59.117 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:07:59.134 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2024-03-24T11:07:59.297 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:07:59.354 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:07:59.409 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:07:59.446 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:07:59.496 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:07:59.598 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:07:59.700 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:56 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:07:59.706 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:07:59.756 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:07:59.852 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:07:59.853 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:59.853 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-24T11:07:59.853 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 137 k 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 137 k 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 379 k 2024-03-24T11:07:59.854 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:00.162 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-17.2.7-726.g9c4e0c3 447 kB/s | 137 kB 00:00 2024-03-24T11:08:00.163 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:00.163 INFO:teuthology.orchestra.run.smithi105.stdout:Total 445 kB/s | 137 kB 00:00 2024-03-24T11:08:00.163 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:00.171 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:00.171 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:00.221 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:00.221 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:00.356 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:00.421 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-24T11:08:00.446 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-24T11:08:00.446 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:08:00.446 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-03-24T11:08:00.446 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:00.470 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:08:00.520 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:08:00.562 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:08:00.616 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:08:00.640 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 1/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:08:01.374 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:08:01.375 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:08:01.375 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:08:01.375 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:08:01.375 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:08:01.375 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:08:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.e 1/1 2024-03-24T11:08:01.385 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:01.385 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:01.386 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:01.386 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:01.666 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2024-03-24T11:08:01.741 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:08:01.741 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:01.741 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:01.741 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:01.741 INFO:teuthology.orchestra.run.smithi100.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:08:01.741 INFO:teuthology.orchestra.run.smithi100.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:01.742 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:01.962 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:08:02.238 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:59 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:02.333 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:02.384 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:02.385 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:02.385 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:02.451 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2024-03-24T11:08:02.555 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:18 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:02.707 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:02.707 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-03-24T11:08:02.707 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 123 k 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:02.709 INFO:teuthology.orchestra.run.smithi100.stdout:======================================================================================= 2024-03-24T11:08:02.710 INFO:teuthology.orchestra.run.smithi100.stdout:Install 6 Packages 2024-03-24T11:08:02.710 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:02.710 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 6.8 M 2024-03-24T11:08:02.710 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 33 M 2024-03-24T11:08:02.710 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:03.000 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:59 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:03.148 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 1.7 M 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 23 k 2024-03-24T11:08:03.149 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 15 k 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout:Install 7 Packages 2024-03-24T11:08:03.150 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:03.151 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.0 M 2024-03-24T11:08:03.151 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 8.1 M 2024-03-24T11:08:03.151 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:03.280 INFO:teuthology.orchestra.run.smithi100.stdout:(1/6): python3-babel-2.9.1-2.el9.noarch.rpm 24 MB/s | 5.8 MB 00:00 2024-03-24T11:08:03.381 INFO:teuthology.orchestra.run.smithi100.stdout:(2/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 728 kB/s | 248 kB 00:00 2024-03-24T11:08:03.489 INFO:teuthology.orchestra.run.smithi100.stdout:(3/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.6 MB/s | 548 kB 00:00 2024-03-24T11:08:03.500 INFO:teuthology.orchestra.run.smithi105.stdout:(1/7): ceph-prometheus-alerts-17.2.7-726.g9c4e0 48 kB/s | 15 kB 00:00 2024-03-24T11:08:03.506 INFO:teuthology.orchestra.run.smithi100.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 460 kB/s | 58 kB 00:00 2024-03-24T11:08:03.531 INFO:teuthology.orchestra.run.smithi100.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 1.2 MB/s | 51 kB 00:00 2024-03-24T11:08:03.566 INFO:teuthology.orchestra.run.smithi105.stdout:(2/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 136 kB/s | 9.0 kB 00:00 2024-03-24T11:08:03.592 INFO:teuthology.orchestra.run.smithi105.stdout:(3/7): ceph-grafana-dashboards-17.2.7-726.g9c4e 59 kB/s | 23 kB 00:00 2024-03-24T11:08:03.634 INFO:teuthology.orchestra.run.smithi105.stdout:(4/7): python3-jwt-2.4.0-1.el9.noarch.rpm 609 kB/s | 41 kB 00:00 2024-03-24T11:08:03.656 INFO:teuthology.orchestra.run.smithi100.stdout:(6/6): ceph-mgr-cephadm-17.2.7-726.g9c4e0c34.el 200 kB/s | 123 kB 00:00 2024-03-24T11:08:03.658 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:03.658 INFO:teuthology.orchestra.run.smithi100.stdout:Total 7.1 MB/s | 6.8 MB 00:00 2024-03-24T11:08:03.670 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:03.680 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:03.680 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:03.684 INFO:teuthology.orchestra.run.smithi105.stdout:(5/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 334 kB/s | 31 kB 00:00 2024-03-24T11:08:03.734 INFO:teuthology.orchestra.run.smithi105.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 1.8 MB/s | 188 kB 00:00 2024-03-24T11:08:03.769 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:03.769 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:03.876 INFO:teuthology.orchestra.run.smithi105.stdout:(7/7): ceph-mgr-dashboard-17.2.7-726.g9c4e0c34. 2.5 MB/s | 1.7 MB 00:00 2024-03-24T11:08:03.878 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:03.878 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.7 MB/s | 2.0 MB 00:00 2024-03-24T11:08:03.878 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:03.886 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:03.886 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:03.952 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:03.952 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:04.059 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:04.221 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:08:04.240 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:04.275 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:08:04.311 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-03-24T11:08:04.345 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-03-24T11:08:04.392 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-03-24T11:08:04.423 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-03-24T11:08:04.452 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 5/7 2024-03-24T11:08:04.681 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 6/7 2024-03-24T11:08:04.705 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-24T11:08:04.785 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:08:04.869 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:08:04.924 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:08:04.954 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-24T11:08:05.228 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 7/7 2024-03-24T11:08:05.228 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.n 1/7 2024-03-24T11:08:05.228 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2/7 2024-03-24T11:08:05.229 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.no 3/7 2024-03-24T11:08:05.229 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-03-24T11:08:05.229 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-03-24T11:08:05.229 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-03-24T11:08:05.545 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:05.546 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:05.778 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2024-03-24T11:08:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-24T11:08:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/6 2024-03-24T11:08:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:08:05.802 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:08:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:08:05.803 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:08:06.099 INFO:teuthology.orchestra.run.smithi100.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:08:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:08:06.100 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:06.100 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:06.335 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:03 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:06.379 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install ceph-fuse 2024-03-24T11:08:06.487 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:06.488 INFO:teuthology.orchestra.run.smithi105.stdout:=========================================================================================== 2024-03-24T11:08:06.488 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:08:06.488 INFO:teuthology.orchestra.run.smithi105.stdout:=========================================================================================== 2024-03-24T11:08:06.488 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:06.488 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 7.4 M 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 CentOS-AppStream 32 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 CentOS-AppStream 3.0 M 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 CentOS-AppStream 17 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 CentOS-AppStream 34 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 CentOS-AppStream 4.7 M 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.17-2.el9 appstream 245 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 CentOS-AppStream 4.8 M 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 CentOS-AppStream 217 k 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 CentOS-AppStream 16 M 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:06.489 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:06.490 INFO:teuthology.orchestra.run.smithi105.stdout:=========================================================================================== 2024-03-24T11:08:06.490 INFO:teuthology.orchestra.run.smithi105.stdout:Install 12 Packages 2024-03-24T11:08:06.490 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:06.490 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 37 M 2024-03-24T11:08:06.490 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 209 M 2024-03-24T11:08:06.491 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:06.725 INFO:teuthology.orchestra.run.smithi105.stdout:(1/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 32 kB 00:00 2024-03-24T11:08:06.759 INFO:teuthology.orchestra.run.smithi105.stdout:(2/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 540 kB/s | 17 kB 00:00 2024-03-24T11:08:06.800 INFO:teuthology.orchestra.run.smithi105.stdout:(3/12): openblas-0.3.21-2.el9.x86_64.rpm 840 kB/s | 34 kB 00:00 2024-03-24T11:08:06.851 INFO:teuthology.orchestra.run.smithi105.stdout:(4/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 11 MB/s | 3.0 MB 00:00 2024-03-24T11:08:06.935 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:22 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:07.035 INFO:teuthology.orchestra.run.smithi105.stdout:(5/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 20 MB/s | 4.7 MB 00:00 2024-03-24T11:08:07.084 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:07.085 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:07.085 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repository Size 2024-03-24T11:08:07.085 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:07.085 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 812 k 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:Installing dependencies: 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:Install 2 Packages 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 891 k 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 2.6 M 2024-03-24T11:08:07.086 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:07.110 INFO:teuthology.orchestra.run.smithi105.stdout:(6/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 19 MB/s | 4.8 MB 00:00 2024-03-24T11:08:07.135 INFO:teuthology.orchestra.run.smithi105.stdout:(7/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 2.1 MB/s | 217 kB 00:00 2024-03-24T11:08:07.242 INFO:teuthology.orchestra.run.smithi100.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 504 kB/s | 79 kB 00:00 2024-03-24T11:08:07.336 INFO:teuthology.orchestra.run.smithi105.stdout:(8/12): ceph-mgr-diskprediction-local-17.2.7-72 9.8 MB/s | 7.4 MB 00:00 2024-03-24T11:08:07.494 INFO:teuthology.orchestra.run.smithi105.stdout:(9/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 41 MB/s | 16 MB 00:00 2024-03-24T11:08:07.567 INFO:teuthology.orchestra.run.smithi100.stdout:(2/2): ceph-fuse-17.2.7-726.g9c4e0c34.el9.x86_6 1.6 MB/s | 812 kB 00:00 2024-03-24T11:08:07.568 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:07.569 INFO:teuthology.orchestra.run.smithi100.stdout:Total 1.8 MB/s | 891 kB 00:00 2024-03-24T11:08:07.569 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:07.589 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:07.589 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:07.619 INFO:teuthology.orchestra.run.smithi105.stdout:(10/12): libgfortran-11.4.1-3.el9.x86_64.rpm 1.6 MB/s | 803 kB 00:00 2024-03-24T11:08:07.645 INFO:teuthology.orchestra.run.smithi105.stdout:(11/12): python3-devel-3.9.17-2.el9.x86_64.rpm 1.6 MB/s | 245 kB 00:00 2024-03-24T11:08:07.659 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:07.660 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:07.678 INFO:teuthology.orchestra.run.smithi105.stdout:(12/12): libquadmath-11.4.1-3.el9.x86_64.rpm 574 kB/s | 195 kB 00:00 2024-03-24T11:08:07.680 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:07.680 INFO:teuthology.orchestra.run.smithi105.stdout:Total 31 MB/s | 37 MB 00:01 2024-03-24T11:08:07.726 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:07.751 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:07.751 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:07.829 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:07.978 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:08:08.041 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-24T11:08:08.190 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:08.190 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-24T11:08:08.703 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/2 2024-03-24T11:08:08.914 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:08.991 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:08:08.991 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:08.991 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:08.991 INFO:teuthology.orchestra.run.smithi100.stdout: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:08:08.991 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:08.991 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:09.005 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 1/12 2024-03-24T11:08:09.032 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-03-24T11:08:09.125 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:08:09.169 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.17-2.el9.x86_64 4/12 2024-03-24T11:08:09.272 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados-devel 2024-03-24T11:08:09.346 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:08:09.376 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:08:09.452 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-03-24T11:08:09.837 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:09.991 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 126 k 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:08:09.992 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:09.993 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 126 k 2024-03-24T11:08:09.993 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 448 k 2024-03-24T11:08:09.993 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:10.103 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-03-24T11:08:10.189 INFO:teuthology.orchestra.run.smithi100.stdout:librados-devel-17.2.7-726.g9c4e0c34.el9.x86_64. 644 kB/s | 126 kB 00:00 2024-03-24T11:08:10.190 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:10.190 INFO:teuthology.orchestra.run.smithi100.stdout:Total 639 kB/s | 126 kB 00:00 2024-03-24T11:08:10.190 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:10.195 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:10.195 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:10.214 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-03-24T11:08:10.249 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:10.249 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:10.394 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:10.495 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:10.936 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:11.258 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:11.258 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:11.258 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:11.258 INFO:teuthology.orchestra.run.smithi100.stdout: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:11.258 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:11.258 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:11.481 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-03-24T11:08:11.543 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs2 2024-03-24T11:08:12.114 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:12.213 INFO:teuthology.orchestra.run.smithi100.stdout:Package libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:12.267 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:12.268 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:12.269 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:12.335 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install libcephfs-devel 2024-03-24T11:08:12.504 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-03-24T11:08:12.526 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-24T11:08:12.900 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:13.047 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:13.048 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-24T11:08:13.048 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:13.048 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-24T11:08:13.048 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 25 k 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 110 k 2024-03-24T11:08:13.049 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:13.210 INFO:teuthology.orchestra.run.smithi100.stdout:libcephfs-devel-17.2.7-726.g9c4e0c34.el9.x86_64 155 kB/s | 25 kB 00:00 2024-03-24T11:08:13.210 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:13.211 INFO:teuthology.orchestra.run.smithi100.stdout:Total 154 kB/s | 25 kB 00:00 2024-03-24T11:08:13.211 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:13.214 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:13.214 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:13.215 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:13.216 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:13.300 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:13.399 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:13.797 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:14.065 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:14.065 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:14.065 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:14.065 INFO:teuthology.orchestra.run.smithi100.stdout: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:14.065 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:14.065 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:14.228 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librados2 2024-03-24T11:08:14.786 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:30 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:14.883 INFO:teuthology.orchestra.run.smithi100.stdout:Package librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:14.937 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:14.938 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:14.938 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:15.003 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install librbd1 2024-03-24T11:08:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 12/12 2024-03-24T11:08:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c 1/12 2024-03-24T11:08:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 2/12 2024-03-24T11:08:15.252 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 3/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 4/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 5/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 7/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 8/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 9/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 10/12 2024-03-24T11:08:15.253 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 11/12 2024-03-24T11:08:15.556 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:15.652 INFO:teuthology.orchestra.run.smithi100.stdout:Package librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.17-2.el9.x86_64 12/12 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-03-24T11:08:15.663 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.17-2.el9.x86_64 2024-03-24T11:08:15.664 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-03-24T11:08:15.664 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-03-24T11:08:15.664 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-03-24T11:08:15.664 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:15.664 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:15.703 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:15.704 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:15.704 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:15.772 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rados 2024-03-24T11:08:16.093 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2024-03-24T11:08:16.322 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:16.418 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:16.469 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:16.470 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:16.470 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:16.535 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rgw 2024-03-24T11:08:16.663 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:13 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:16.813 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 47 k 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.28.2-1.el9 epel 203 k 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 CentOS-AppStream 24 k 2024-03-24T11:08:16.814 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 CentOS-AppStream 18 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 CentOS-AppStream 174 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 CentOS-AppStream 132 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 CentOS-AppStream 211 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 CentOS-AppStream 43 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2024-03-24T11:08:16.815 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:16.816 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.0 M 2024-03-24T11:08:16.816 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-03-24T11:08:16.816 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:17.018 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 183 kB/s | 24 kB 00:00 2024-03-24T11:08:17.044 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 114 kB/s | 18 kB 00:00 2024-03-24T11:08:17.090 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:33 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:17.102 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 2.3 MB/s | 132 kB 00:00 2024-03-24T11:08:17.127 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 2.1 MB/s | 174 kB 00:00 2024-03-24T11:08:17.153 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-pyasn1-modules-0.4.8-6.el9.noar 4.1 MB/s | 211 kB 00:00 2024-03-24T11:08:17.178 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 852 kB/s | 43 kB 00:00 2024-03-24T11:08:17.185 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:17.228 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): ceph-mgr-rook-17.2.7-726.g9c4e0c34.el9. 138 kB/s | 47 kB 00:00 2024-03-24T11:08:17.237 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:17.238 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:17.238 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:17.301 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-cephfs 2024-03-24T11:08:17.420 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 120 kB/s | 32 kB 00:00 2024-03-24T11:08:17.446 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 56 kB/s | 15 kB 00:00 2024-03-24T11:08:17.587 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 417 kB/s | 59 kB 00:00 2024-03-24T11:08:17.621 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-google-auth-2.28.2-1.el9.noarc 585 kB/s | 203 kB 00:00 2024-03-24T11:08:17.705 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 767 kB/s | 90 kB 00:00 2024-03-24T11:08:17.746 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 3.1 MB/s | 1.0 MB 00:00 2024-03-24T11:08:17.750 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:17.750 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.2 MB/s | 2.0 MB 00:00 2024-03-24T11:08:17.751 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:17.763 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:17.764 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:17.852 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:17.852 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:17.862 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:33 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:17.959 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:18.010 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:18.011 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:18.012 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:18.075 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install python3-rbd 2024-03-24T11:08:18.357 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:18.527 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-03-24T11:08:18.581 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-03-24T11:08:18.631 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:18.637 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-03-24T11:08:18.674 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-03-24T11:08:18.715 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-03-24T11:08:18.727 INFO:teuthology.orchestra.run.smithi100.stdout:Package python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:18.779 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:18.780 INFO:teuthology.orchestra.run.smithi100.stdout:Nothing to do. 2024-03-24T11:08:18.780 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:18.817 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-03-24T11:08:18.845 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-fuse 2024-03-24T11:08:18.934 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.28.2-1.el9.noarch 7/13 2024-03-24T11:08:18.984 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-03-24T11:08:19.398 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:19.549 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:19.550 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:19.550 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-03-24T11:08:19.550 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:19.551 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:19.551 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 82 k 2024-03-24T11:08:19.551 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:19.551 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:19.551 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:19.551 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:08:19.552 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:19.552 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 82 k 2024-03-24T11:08:19.552 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 222 k 2024-03-24T11:08:19.552 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:19.705 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-03-24T11:08:19.754 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-fuse-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 401 kB/s | 82 kB 00:00 2024-03-24T11:08:19.755 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:19.755 INFO:teuthology.orchestra.run.smithi100.stdout:Total 398 kB/s | 82 kB 00:00 2024-03-24T11:08:19.755 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:19.756 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-03-24T11:08:19.762 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:19.762 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:19.798 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-03-24T11:08:19.816 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:19.816 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:19.852 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-03-24T11:08:19.876 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-24T11:08:19.954 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:20.060 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:20.452 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:20.713 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 13/13 2024-03-24T11:08:20.713 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 1/13 2024-03-24T11:08:20.713 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-03-24T11:08:20.713 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-03-24T11:08:20.713 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.28.2-1.el9.noarch 10/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-03-24T11:08:20.714 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-03-24T11:08:20.747 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:20.747 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:20.747 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:20.748 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:20.748 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:20.748 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:21.000 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-mirror 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.28.2-1.el9.noarch 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-03-24T11:08:21.061 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:21.062 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:21.323 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2024-03-24T11:08:21.561 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:21.712 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:21.713 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:21.713 INFO:teuthology.orchestra.run.smithi100.stdout: Package Arch Version Repo Size 2024-03-24T11:08:21.713 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 2.9 M 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 12 M 2024-03-24T11:08:21.714 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:21.882 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:18 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:22.030 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:17.2.7-726.g9c4e0c34.el9 ceph-noarch 123 k 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-03-24T11:08:22.031 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 CentOS-AppStream 5.8 M 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-5.el9 appstream 248 k 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-03-24T11:08:22.032 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:22.033 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 6.8 M 2024-03-24T11:08:22.033 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 33 M 2024-03-24T11:08:22.033 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:22.085 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-mirror-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 7.9 MB/s | 2.9 MB 00:00 2024-03-24T11:08:22.086 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:22.086 INFO:teuthology.orchestra.run.smithi100.stdout:Total 7.9 MB/s | 2.9 MB 00:00 2024-03-24T11:08:22.086 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:22.095 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:22.095 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:22.168 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:22.168 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:22.515 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): python3-jinja2-2.11.3-5.el9.noarch.rpm 1.4 MB/s | 248 kB 00:00 2024-03-24T11:08:22.576 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:22.624 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): python3-babel-2.9.1-2.el9.noarch.rpm 20 MB/s | 5.8 MB 00:00 2024-03-24T11:08:22.644 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:22.658 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): ceph-mgr-cephadm-17.2.7-726.g9c4e0c34.el 388 kB/s | 123 kB 00:00 2024-03-24T11:08:22.669 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:22.670 INFO:teuthology.orchestra.run.smithi100.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:08:22.670 INFO:teuthology.orchestra.run.smithi100.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:08:22.670 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:08:22.670 INFO:teuthology.orchestra.run.smithi100.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:08:22.670 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:22.725 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): python3-natsort-7.1.1-5.el9.noarch.rpm 574 kB/s | 58 kB 00:00 2024-03-24T11:08:22.750 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 555 kB/s | 51 kB 00:00 2024-03-24T11:08:22.786 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 2.0 MB/s | 548 kB 00:00 2024-03-24T11:08:22.786 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:22.786 INFO:teuthology.orchestra.run.smithi105.stdout:Total 9.0 MB/s | 6.8 MB 00:00 2024-03-24T11:08:22.799 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:22.809 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:22.809 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:22.896 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:22.897 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:23.198 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:23.364 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-03-24T11:08:23.418 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-03-24T11:08:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:23.880 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:23.880 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:23.881 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:23.881 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:23.933 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-03-24T11:08:24.028 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-03-24T11:08:24.092 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-5.el9.noarch 5/6 2024-03-24T11:08:24.123 DEBUG:teuthology.orchestra.run.smithi100:> sudo yum -y install rbd-nbd 2024-03-24T11:08:24.123 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-24T11:08:24.693 INFO:teuthology.orchestra.run.smithi100.stdout:Last metadata expiration check: 0:01:40 ago on Sun 24 Mar 2024 11:06:44 AM UTC. 2024-03-24T11:08:24.841 INFO:teuthology.orchestra.run.smithi100.stdout:Dependencies resolved. 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout: Package Architecture Version Repository Size 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout:Installing: 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 165 k 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction Summary 2024-03-24T11:08:24.842 INFO:teuthology.orchestra.run.smithi100.stdout:================================================================================ 2024-03-24T11:08:24.843 INFO:teuthology.orchestra.run.smithi100.stdout:Install 1 Package 2024-03-24T11:08:24.843 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:24.843 INFO:teuthology.orchestra.run.smithi100.stdout:Total download size: 165 k 2024-03-24T11:08:24.843 INFO:teuthology.orchestra.run.smithi100.stdout:Installed size: 477 k 2024-03-24T11:08:24.843 INFO:teuthology.orchestra.run.smithi100.stdout:Downloading Packages: 2024-03-24T11:08:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 6/6 2024-03-24T11:08:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 1/6 2024-03-24T11:08:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-03-24T11:08:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-5.el9.noarch 3/6 2024-03-24T11:08:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-03-24T11:08:24.923 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-03-24T11:08:25.055 INFO:teuthology.orchestra.run.smithi100.stdout:rbd-nbd-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 777 kB/s | 165 kB 00:00 2024-03-24T11:08:25.056 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:25.056 INFO:teuthology.orchestra.run.smithi100.stdout:Total 772 kB/s | 165 kB 00:00 2024-03-24T11:08:25.056 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction check 2024-03-24T11:08:25.064 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction check succeeded. 2024-03-24T11:08:25.064 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction test 2024-03-24T11:08:25.119 INFO:teuthology.orchestra.run.smithi100.stdout:Transaction test succeeded. 2024-03-24T11:08:25.119 INFO:teuthology.orchestra.run.smithi100.stdout:Running transaction 2024-03-24T11:08:25.225 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-03-24T11:08:25.225 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:25.225 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:25.225 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:17.2.7-726.g9c4e0c34.el9.noarch 2024-03-24T11:08:25.225 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-03-24T11:08:25.226 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-03-24T11:08:25.226 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-5.el9.noarch 2024-03-24T11:08:25.226 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-03-24T11:08:25.226 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-03-24T11:08:25.226 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:25.226 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:25.273 INFO:teuthology.orchestra.run.smithi100.stdout: Preparing : 1/1 2024-03-24T11:08:25.394 INFO:teuthology.orchestra.run.smithi100.stdout: Installing : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:25.490 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2024-03-24T11:08:25.857 INFO:teuthology.orchestra.run.smithi100.stdout: Running scriptlet: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:26.047 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:23 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: Verifying : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:26.137 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:26.138 INFO:teuthology.orchestra.run.smithi100.stdout:Installed: 2024-03-24T11:08:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:26.138 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:08:26.138 INFO:teuthology.orchestra.run.smithi100.stdout:Complete! 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 812 k 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-03-24T11:08:26.197 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 CentOS-BaseOS 79 k 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 891 k 2024-03-24T11:08:26.198 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.6 M 2024-03-24T11:08:26.199 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:26.327 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 612 kB/s | 79 kB 00:00 2024-03-24T11:08:26.393 DEBUG:teuthology.parallel:result is None 2024-03-24T11:08:26.502 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-fuse-17.2.7-726.g9c4e0c34.el9.x86_6 2.6 MB/s | 812 kB 00:00 2024-03-24T11:08:26.503 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:26.504 INFO:teuthology.orchestra.run.smithi105.stdout:Total 2.8 MB/s | 891 kB 00:00 2024-03-24T11:08:26.504 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:26.524 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:26.524 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:26.594 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:26.595 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:26.767 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:26.905 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-03-24T11:08:26.976 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-24T11:08:27.704 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2/2 2024-03-24T11:08:27.704 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/2 2024-03-24T11:08:27.992 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-03-24T11:08:27.992 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:27.992 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:27.993 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-03-24T11:08:27.993 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:27.993 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:28.281 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2024-03-24T11:08:28.857 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:25 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:29.008 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 126 k 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:08:29.009 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:29.010 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 126 k 2024-03-24T11:08:29.010 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 448 k 2024-03-24T11:08:29.010 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:29.232 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-17.2.7-726.g9c4e0c34.el9.x86_64. 569 kB/s | 126 kB 00:00 2024-03-24T11:08:29.233 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:29.233 INFO:teuthology.orchestra.run.smithi105.stdout:Total 565 kB/s | 126 kB 00:00 2024-03-24T11:08:29.233 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:29.238 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:29.238 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:29.292 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:29.292 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:29.442 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:29.547 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:30.040 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:30.320 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:30.320 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:30.320 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:30.320 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:30.320 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:30.320 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:30.655 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2024-03-24T11:08:31.230 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:28 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:31.326 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:31.377 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:31.378 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:31.457 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2024-03-24T11:08:32.019 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:29 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:32.164 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 25 k 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:32.165 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:08:32.166 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:32.166 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 25 k 2024-03-24T11:08:32.166 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 110 k 2024-03-24T11:08:32.166 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:32.322 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-17.2.7-726.g9c4e0c34.el9.x86_64 160 kB/s | 25 kB 00:00 2024-03-24T11:08:32.322 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:32.322 INFO:teuthology.orchestra.run.smithi105.stdout:Total 159 kB/s | 25 kB 00:00 2024-03-24T11:08:32.323 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:32.326 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:32.326 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:32.327 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:32.327 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:32.412 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:32.510 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:32.914 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:33.174 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:33.174 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:33.175 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:33.175 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:33.175 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:33.175 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:33.370 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2024-03-24T11:08:33.925 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:30 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:34.022 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:34.073 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:34.074 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:34.074 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:34.156 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2024-03-24T11:08:34.718 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:31 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:34.815 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:34.868 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:34.869 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:34.869 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:34.951 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2024-03-24T11:08:35.510 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:32 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:35.607 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:35.659 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:35.660 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:35.660 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:35.751 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2024-03-24T11:08:36.306 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:33 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:36.402 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:36.454 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:36.456 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:36.456 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:36.531 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2024-03-24T11:08:37.091 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:37.189 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:37.241 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:37.242 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:37.242 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:37.324 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2024-03-24T11:08:37.877 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:34 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:37.973 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 is already installed. 2024-03-24T11:08:38.028 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:38.029 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-03-24T11:08:38.029 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:38.117 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2024-03-24T11:08:38.678 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:35 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:38.827 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 82 k 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:38.828 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:38.829 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:08:38.829 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:38.829 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 82 k 2024-03-24T11:08:38.829 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 222 k 2024-03-24T11:08:38.829 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:39.141 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 261 kB/s | 82 kB 00:00 2024-03-24T11:08:39.142 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:39.143 INFO:teuthology.orchestra.run.smithi105.stdout:Total 260 kB/s | 82 kB 00:00 2024-03-24T11:08:39.143 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:39.150 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:39.150 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:39.203 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:39.203 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:39.336 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:39.439 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:39.816 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:40.081 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:40.081 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:40.081 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:40.081 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:40.081 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:40.081 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:40.293 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2024-03-24T11:08:40.864 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:37 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:41.024 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:41.024 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:41.024 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-03-24T11:08:41.024 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:41.024 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:41.024 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 2.9 M 2024-03-24T11:08:41.025 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:41.025 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:41.025 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:41.025 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:08:41.025 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:41.026 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.9 M 2024-03-24T11:08:41.026 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 12 M 2024-03-24T11:08:41.026 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:41.728 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 4.2 MB/s | 2.9 MB 00:00 2024-03-24T11:08:41.729 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:41.729 INFO:teuthology.orchestra.run.smithi105.stdout:Total 4.2 MB/s | 2.9 MB 00:00 2024-03-24T11:08:41.729 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:41.738 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:41.738 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:41.810 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:41.810 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:42.212 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:42.279 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:42.303 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:42.303 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-03-24T11:08:42.304 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-03-24T11:08:42.304 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:08:42.304 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-03-24T11:08:42.304 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:43.396 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:43.396 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:43.396 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:43.396 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:43.396 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:43.396 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:43.632 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2024-03-24T11:08:44.199 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:41 ago on Sun 24 Mar 2024 11:07:03 AM UTC. 2024-03-24T11:08:44.347 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:17.2.7-726.g9c4e0c34.el9 ceph 165 k 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 165 k 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 477 k 2024-03-24T11:08:44.348 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-03-24T11:08:44.690 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-17.2.7-726.g9c4e0c34.el9.x86_64.rpm 484 kB/s | 165 kB 00:00 2024-03-24T11:08:44.691 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:08:44.691 INFO:teuthology.orchestra.run.smithi105.stdout:Total 482 kB/s | 165 kB 00:00 2024-03-24T11:08:44.691 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-03-24T11:08:44.698 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-03-24T11:08:44.698 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-03-24T11:08:44.752 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-03-24T11:08:44.752 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-03-24T11:08:44.895 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-03-24T11:08:44.987 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:45.470 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:45.755 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 1/1 2024-03-24T11:08:45.755 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:45.756 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-03-24T11:08:45.756 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:17.2.7-726.g9c4e0c34.el9.x86_64 2024-03-24T11:08:45.756 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:08:45.756 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-03-24T11:08:46.035 DEBUG:teuthology.parallel:result is None 2024-03-24T11:08:46.036 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:08:46.036 INFO:teuthology.packaging:ref: None 2024-03-24T11:08:46.036 INFO:teuthology.packaging:tag: None 2024-03-24T11:08:46.036 INFO:teuthology.packaging:branch: quincy 2024-03-24T11:08:46.036 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:08:46.036 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-24T11:08:46.273 DEBUG:teuthology.orchestra.run.smithi100:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:08:46.298 INFO:teuthology.orchestra.run.smithi100.stdout:17.2.7-726.g9c4e0c34.el9 2024-03-24T11:08:46.298 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-726.g9c4e0c34.el9 2024-03-24T11:08:46.299 INFO:teuthology.task.install:The correct ceph version 17.2.7-726.g9c4e0c34 is installed. 2024-03-24T11:08:46.300 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-03-24T11:08:46.300 INFO:teuthology.packaging:ref: None 2024-03-24T11:08:46.300 INFO:teuthology.packaging:tag: None 2024-03-24T11:08:46.300 INFO:teuthology.packaging:branch: quincy 2024-03-24T11:08:46.300 INFO:teuthology.packaging:sha1: db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:08:46.301 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-03-24T11:08:46.687 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-03-24T11:08:46.710 INFO:teuthology.orchestra.run.smithi105.stdout:17.2.7-726.g9c4e0c34.el9 2024-03-24T11:08:46.710 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-726.g9c4e0c34.el9 2024-03-24T11:08:46.711 INFO:teuthology.task.install:The correct ceph version 17.2.7-726.g9c4e0c34 is installed. 2024-03-24T11:08:46.714 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-03-24T11:08:46.715 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:08:46.715 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:08:46.748 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:08:46.748 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-03-24T11:08:46.779 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-03-24T11:08:46.780 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:08:46.780 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:08:46.819 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:08:46.889 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:08:46.889 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-03-24T11:08:46.917 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-03-24T11:08:46.985 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-03-24T11:08:46.985 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:08:46.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:08:47.014 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:08:47.082 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:08:47.082 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-03-24T11:08:47.109 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-03-24T11:08:47.176 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-03-24T11:08:47.176 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:08:47.177 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:08:47.203 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:08:47.271 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:08:47.271 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-03-24T11:08:47.298 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-03-24T11:08:47.365 INFO:teuthology.run_tasks:Running task print... 2024-03-24T11:08:47.374 INFO:teuthology.task.print:**** done install task... 2024-03-24T11:08:47.375 INFO:teuthology.run_tasks:Running task cephadm... 2024-03-24T11:08:47.562 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'db0330b1e4e2470d52b750e251e55a522b4f7d69'} 2024-03-24T11:08:47.562 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-03-24T11:08:47.563 INFO:tasks.cephadm:Cluster fsid is e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:08:47.563 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-03-24T11:08:47.563 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-03-24T11:08:47.563 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi100': '172.21.15.100', 'mon.smithi105': '172.21.15.105'} 2024-03-24T11:08:47.563 INFO:tasks.cephadm:Normalizing hostnames... 2024-03-24T11:08:47.563 DEBUG:teuthology.orchestra.run.smithi100:> sudo hostname $(hostname -s) 2024-03-24T11:08:47.591 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-03-24T11:08:47.618 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-03-24T11:08:47.619 DEBUG:teuthology.orchestra.run.smithi100:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:08:47.832 INFO:teuthology.orchestra.run.smithi100.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 24 11:08 /home/ubuntu/cephtest/cephadm 2024-03-24T11:08:47.832 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-03-24T11:08:47.996 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Mar 24 11:08 /home/ubuntu/cephtest/cephadm 2024-03-24T11:08:47.997 DEBUG:teuthology.orchestra.run.smithi100:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:08:48.011 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-03-24T11:08:48.037 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-03-24T11:08:48.037 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-24T11:08:48.054 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-03-24T11:08:48.922 INFO:teuthology.orchestra.run.smithi100.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-24T11:08:48.926 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-24T11:09:29.398 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-03-24T11:09:29.398 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-24T11:09:29.398 INFO:teuthology.orchestra.run.smithi100.stdout: "image_id": "837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de", 2024-03-24T11:09:29.399 INFO:teuthology.orchestra.run.smithi100.stdout: "repo_digests": [ 2024-03-24T11:09:29.399 INFO:teuthology.orchestra.run.smithi100.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232" 2024-03-24T11:09:29.399 INFO:teuthology.orchestra.run.smithi100.stdout: ] 2024-03-24T11:09:29.399 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-03-24T11:09:33.617 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-03-24T11:09:33.618 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)", 2024-03-24T11:09:33.618 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de", 2024-03-24T11:09:33.618 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-03-24T11:09:33.618 INFO:teuthology.orchestra.run.smithi105.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232" 2024-03-24T11:09:33.618 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-03-24T11:09:33.618 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-03-24T11:09:33.638 DEBUG:teuthology.orchestra.run.smithi100:> sudo mkdir -p /etc/ceph 2024-03-24T11:09:33.677 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-03-24T11:09:33.707 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 777 /etc/ceph 2024-03-24T11:09:33.747 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-03-24T11:09:33.774 INFO:tasks.cephadm:Writing seed config... 2024-03-24T11:09:33.775 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-03-24T11:09:33.776 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [client] debug client = 20 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-03-24T11:09:33.777 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-03-24T11:09:33.778 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-03-24T11:09:33.779 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:09:33.779 DEBUG:teuthology.orchestra.run.smithi100:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-03-24T11:09:33.804 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log to file = true mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = e2f9ef6c-e9ce-11ee-95cd-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-03-24T11:09:33.804 DEBUG:teuthology.orchestra.run.smithi100:mon.smithi100> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service 2024-03-24T11:09:33.848 INFO:tasks.cephadm:Bootstrapping... 2024-03-24T11:09:33.849 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.100 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:09:34.056 INFO:teuthology.orchestra.run.smithi100.stdout:-------------------------------------------------------------------------------- 2024-03-24T11:09:34.056 INFO:teuthology.orchestra.run.smithi100.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'e2f9ef6c-e9ce-11ee-95cd-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.100', '--skip-admin-label'] 2024-03-24T11:09:34.081 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:09:34.081 INFO:teuthology.orchestra.run.smithi100.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-03-24T11:09:34.082 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying podman|docker is present... 2024-03-24T11:09:34.105 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:09:34.105 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying lvm2 is present... 2024-03-24T11:09:34.105 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying time synchronization is in place... 2024-03-24T11:09:34.113 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:09:34.113 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:09:34.119 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:09:34.119 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-03-24T11:09:34.126 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout enabled 2024-03-24T11:09:34.132 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout active 2024-03-24T11:09:34.132 INFO:teuthology.orchestra.run.smithi100.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:09:34.132 INFO:teuthology.orchestra.run.smithi100.stdout:Repeating the final host check... 2024-03-24T11:09:34.155 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 4.9.3-dev 2024-03-24T11:09:34.155 INFO:teuthology.orchestra.run.smithi100.stdout:podman (/bin/podman) version 4.9.3 is present 2024-03-24T11:09:34.156 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl is present 2024-03-24T11:09:34.156 INFO:teuthology.orchestra.run.smithi100.stdout:lvcreate is present 2024-03-24T11:09:34.162 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-03-24T11:09:34.162 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-03-24T11:09:34.168 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-03-24T11:09:34.168 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-03-24T11:09:34.174 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout enabled 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout active 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:Unit chronyd.service is enabled and running 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:Host looks OK 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:Cluster fsid: e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:Acquiring lock 139885610394720 on /run/cephadm/e2f9ef6c-e9ce-11ee-95cd-87774f69a715.lock 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:Lock 139885610394720 acquired on /run/cephadm/e2f9ef6c-e9ce-11ee-95cd-87774f69a715.lock 2024-03-24T11:09:34.181 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying IP 172.21.15.100 port 3300 ... 2024-03-24T11:09:34.182 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying IP 172.21.15.100 port 6789 ... 2024-03-24T11:09:34.182 INFO:teuthology.orchestra.run.smithi100.stdout:Base mon IP(s) is [172.21.15.100:3300, 172.21.15.100:6789], mon addrv is [v2:172.21.15.100:3300,v1:172.21.15.100:6789] 2024-03-24T11:09:34.185 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.100 metric 100 2024-03-24T11:09:34.185 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.100 metric 100 2024-03-24T11:09:34.188 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-03-24T11:09:34.188 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-03-24T11:09:34.188 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1765sec hoplimit 64 pref medium 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-03-24T11:09:34.191 INFO:teuthology.orchestra.run.smithi100.stdout:Mon IP `172.21.15.100` is in CIDR network `172.21.0.0/20` 2024-03-24T11:09:34.192 INFO:teuthology.orchestra.run.smithi100.stdout:Mon IP `172.21.15.100` is in CIDR network `172.21.0.0/20` 2024-03-24T11:09:34.192 INFO:teuthology.orchestra.run.smithi100.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-03-24T11:09:34.192 INFO:teuthology.orchestra.run.smithi100.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-03-24T11:09:34.192 INFO:teuthology.orchestra.run.smithi100.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-24T11:09:34.747 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stdout 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de 2024-03-24T11:09:34.748 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-03-24T11:09:34.748 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Getting image source signatures 2024-03-24T11:09:34.748 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Copying blob sha256:601ec453fe24458667695dda21742a4f3ea719c31ad8aa8ec48bf585cc42a7d7 2024-03-24T11:09:34.748 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Copying blob sha256:9516588f908cdb14b5e4062a6063db2d9291b355c3f32e2a99864608a4a7b649 2024-03-24T11:09:34.748 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Copying config sha256:837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de 2024-03-24T11:09:34.748 INFO:teuthology.orchestra.run.smithi100.stdout:/bin/podman: stderr Writing manifest to image destination 2024-03-24T11:09:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:ceph: stdout ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable) 2024-03-24T11:09:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:Ceph version: ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable) 2024-03-24T11:09:35.603 INFO:teuthology.orchestra.run.smithi100.stdout:Extracting ceph user uid/gid from container image... 2024-03-24T11:09:36.550 INFO:teuthology.orchestra.run.smithi100.stdout:stat: stdout 167 167 2024-03-24T11:09:36.550 INFO:teuthology.orchestra.run.smithi100.stdout:Creating initial keys... 2024-03-24T11:09:37.431 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph-authtool: stdout AQDwCQBm4I8fMhAAcVC8BPAGDIuZxyzr4qtZyA== 2024-03-24T11:09:38.403 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph-authtool: stdout AQDxCQBmSaF4LhAA9ui3qLbWbfU2yF7jCq2VBg== 2024-03-24T11:09:39.484 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph-authtool: stdout AQDyCQBmzwfOLBAAb2BSi9Zsh3hxvdG40Pj3aQ== 2024-03-24T11:09:39.485 INFO:teuthology.orchestra.run.smithi100.stdout:Creating initial monmap... 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:monmaptool for smithi100 [v2:172.21.15.100:3300,v1:172.21.15.100:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:setting min_mon_release = octopus 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: set fsid to e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-03-24T11:09:40.457 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:09:40.458 INFO:teuthology.orchestra.run.smithi100.stdout:Creating mon... 2024-03-24T11:09:41.354 INFO:teuthology.orchestra.run.smithi100.stdout:create mon.smithi100 on 2024-03-24T11:09:41.548 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-03-24T11:09:41.698 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-03-24T11:09:41.852 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715.target. 2024-03-24T11:09:41.853 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715.target → /etc/systemd/system/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715.target. 2024-03-24T11:09:42.050 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100 2024-03-24T11:09:42.050 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service: Unit ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service not loaded. 2024-03-24T11:09:42.224 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715.target.wants/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service → /etc/systemd/system/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@.service. 2024-03-24T11:09:42.282 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 systemd[1]: Starting Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:09:42.554 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 podman[28552]: 2024-03-24 11:09:42.461879602 +0000 UTC m=+0.110538969 container create 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, GIT_CLEAN=True, io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2) 2024-03-24T11:09:42.554 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 podman[28552]: 2024-03-24 11:09:42.368931884 +0000 UTC m=+0.017591261 image pull 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de quay.ceph.io/ceph-ci/ceph:quincy 2024-03-24T11:09:42.763 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:09:42.764 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-03-24T11:09:42.772 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:09:42.772 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-03-24T11:09:42.772 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-03-24T11:09:42.772 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:09:42.772 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mon to start... 2024-03-24T11:09:42.773 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mon... 2024-03-24T11:09:42.897 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 podman[28552]: 2024-03-24 11:09:42.647710554 +0000 UTC m=+0.296369923 container init 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, GIT_CLEAN=True) 2024-03-24T11:09:42.897 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 podman[28552]: 2024-03-24 11:09:42.650640489 +0000 UTC m=+0.299299851 container start 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, org.label-schema.license=GPLv2, ceph=True, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, RELEASE=quincy-9c4e0c3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5) 2024-03-24T11:09:42.898 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 ceph-mon[28571]: mkfs e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:42.898 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 ceph-mon[28571]: mon.smithi100 is new leader, mons smithi100 in quorum (ranks 0) 2024-03-24T11:09:42.898 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 bash[28552]: 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b 2024-03-24T11:09:42.898 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:42 smithi100 systemd[1]: Started Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout cluster: 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout id: e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout services: 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi100 (age 0.585172s) 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout data: 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-03-24T11:09:43.872 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout pgs: 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.219+0000 7ff7517a1700 1 Processor -- start 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.219+0000 7ff7517a1700 1 -- start start 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.219+0000 7ff7517a1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c1051e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.219+0000 7ff7517a1700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c105720 con 0x7ff74c076b40 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c1051e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c1051e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50910/0 (socket says 172.21.15.100:50910) 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74bfff700 1 -- 172.21.15.100:0/1890558075 learned_addr learned my addr 172.21.15.100:0/1890558075 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74bfff700 1 -- 172.21.15.100:0/1890558075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff74c105860 con 0x7ff74c076b40 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74bfff700 1 --2- 172.21.15.100:0/1890558075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c1051e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff7400098e0 tx=0x7ff740009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c23731d7e279067f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:43.873 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74affd700 1 -- 172.21.15.100:0/1890558075 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74000b860 con 0x7ff74c076b40 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74affd700 1 -- 172.21.15.100:0/1890558075 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74000b9c0 con 0x7ff74c076b40 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.220+0000 7ff74affd700 1 -- 172.21.15.100:0/1890558075 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74000bc90 con 0x7ff74c076b40 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 -- 172.21.15.100:0/1890558075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 msgr2=0x7ff74c1051e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 --2- 172.21.15.100:0/1890558075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c1051e0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7ff7400098e0 tx=0x7ff740009250 comp rx=0 tx=0).stop 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 -- 172.21.15.100:0/1890558075 shutdown_connections 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 --2- 172.21.15.100:0/1890558075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c1051e0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 -- 172.21.15.100:0/1890558075 >> 172.21.15.100:0/1890558075 conn(0x7ff74c0fcb90 msgr2=0x7ff74c0fefc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 -- 172.21.15.100:0/1890558075 shutdown_connections 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.221+0000 7ff7517a1700 1 -- 172.21.15.100:0/1890558075 wait complete. 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff7517a1700 1 Processor -- start 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff7517a1700 1 -- start start 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff7517a1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c110bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff7517a1700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff74c111110 con 0x7ff74c076b40 2024-03-24T11:09:43.874 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff74bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c110bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff74bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c110bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50916/0 (socket says 172.21.15.100:50916) 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff74bfff700 1 -- 172.21.15.100:0/165653133 learned_addr learned my addr 172.21.15.100:0/165653133 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff74bfff700 1 -- 172.21.15.100:0/165653133 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7400094f0 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.222+0000 7ff74bfff700 1 --2- 172.21.15.100:0/165653133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c110bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff740009290 tx=0x7ff740016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74001c5c0 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff74001c740 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff74c111310 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff74001ca50 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff74c111e80 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7ff74001a450 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff74002e790 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.223+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff74c05f5f0 con 0x7ff74c076b40 2024-03-24T11:09:43.875 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.226+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7ff74001ce30 con 0x7ff74c076b40 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.294+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7ff74c112890 con 0x7ff74c076b40 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.294+0000 7ff748ff9700 1 -- 172.21.15.100:0/165653133 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7ff74001a840 con 0x7ff74c076b40 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 msgr2=0x7ff74c110bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 --2- 172.21.15.100:0/165653133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c110bd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7ff740009290 tx=0x7ff740016000 comp rx=0 tx=0).stop 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 shutdown_connections 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 --2- 172.21.15.100:0/165653133 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff74c076b40 0x7ff74c110bd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 >> 172.21.15.100:0/165653133 conn(0x7ff74c0fcb90 msgr2=0x7ff74c0fcfb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 shutdown_connections 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:43.295+0000 7ff7517a1700 1 -- 172.21.15.100:0/165653133 wait complete. 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:mon is available 2024-03-24T11:09:43.876 INFO:teuthology.orchestra.run.smithi100.stdout:Assimilating anything we can from ceph.conf... 2024-03-24T11:09:43.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:43 smithi100 ceph-mon[28571]: mon.smithi100 is new leader, mons smithi100 in quorum (ranks 0) 2024-03-24T11:09:43.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:43 smithi100 ceph-mon[28571]: monmap e1: 1 mons at {smithi100=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:09:43.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:43 smithi100 ceph-mon[28571]: fsmap 2024-03-24T11:09:43.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:43 smithi100 ceph-mon[28571]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:09:43.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:43 smithi100 ceph-mon[28571]: mgrmap e1: no daemons active 2024-03-24T11:09:43.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:43 smithi100 ceph-mon[28571]: from='client.? 172.21.15.100:0/165653133' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-03-24T11:09:44.937 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:44 smithi100 ceph-mon[28571]: from='client.? 172.21.15.100:0/2613200926' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:09:44.937 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:44 smithi100 ceph-mon[28571]: from='client.? 172.21.15.100:0/2613200926' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-03-24T11:09:44.946 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout fsid = e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.100:3300,v1:172.21.15.100:6789] 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:09:44.947 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba7bc47700 1 Processor -- start 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba7bc47700 1 -- start start 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba7bc47700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74077200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba7bc47700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba740790b0 con 0x7fba74078ca0 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba799e3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74077200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba799e3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74077200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50922/0 (socket says 172.21.15.100:50922) 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.279+0000 7fba799e3700 1 -- 172.21.15.100:0/3523410396 learned_addr learned my addr 172.21.15.100:0/3523410396 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba799e3700 1 -- 172.21.15.100:0/3523410396 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba74077740 con 0x7fba74078ca0 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba799e3700 1 --2- 172.21.15.100:0/3523410396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74077200 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fba700098e0 tx=0x7fba70009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4890cf1593d52dbf server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:44.948 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba789e1700 1 -- 172.21.15.100:0/3523410396 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7000b860 con 0x7fba74078ca0 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba789e1700 1 -- 172.21.15.100:0/3523410396 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7000b9c0 con 0x7fba74078ca0 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba789e1700 1 -- 172.21.15.100:0/3523410396 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7000bc90 con 0x7fba74078ca0 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba7bc47700 1 -- 172.21.15.100:0/3523410396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 msgr2=0x7fba74077200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.280+0000 7fba7bc47700 1 --2- 172.21.15.100:0/3523410396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74077200 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fba700098e0 tx=0x7fba70009250 comp rx=0 tx=0).stop 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 -- 172.21.15.100:0/3523410396 shutdown_connections 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 --2- 172.21.15.100:0/3523410396 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74077200 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 -- 172.21.15.100:0/3523410396 >> 172.21.15.100:0/3523410396 conn(0x7fba740fde40 msgr2=0x7fba741002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 -- 172.21.15.100:0/3523410396 shutdown_connections 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 -- 172.21.15.100:0/3523410396 wait complete. 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 Processor -- start 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 -- start start 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74112f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba7bc47700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fba74113490 con 0x7fba74078ca0 2024-03-24T11:09:44.949 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba799e3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74112f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba799e3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74112f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50932/0 (socket says 172.21.15.100:50932) 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.281+0000 7fba799e3700 1 -- 172.21.15.100:0/2613200926 learned_addr learned my addr 172.21.15.100:0/2613200926 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba799e3700 1 -- 172.21.15.100:0/2613200926 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba700094f0 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba799e3700 1 --2- 172.21.15.100:0/2613200926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74112f50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fba70009290 tx=0x7fba70016000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7001c5c0 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fba7001c740 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fba74113690 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba7001ca50 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fba74114200 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fba7001a450 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.282+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fba7002e790 con 0x7fba74078ca0 2024-03-24T11:09:44.950 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.283+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fba7405f5f0 con 0x7fba74078ca0 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.285+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fba7001ce30 con 0x7fba74078ca0 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.350+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fba74077c40 con 0x7fba74078ca0 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.353+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 7 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fba7001b3e0 con 0x7fba74078ca0 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.353+0000 7fba6a7fc700 1 -- 172.21.15.100:0/2613200926 <== mon.0 v2:172.21.15.100:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7fba74077c40 con 0x7fba74078ca0 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 msgr2=0x7fba74112f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 --2- 172.21.15.100:0/2613200926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74112f50 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fba70009290 tx=0x7fba70016000 comp rx=0 tx=0).stop 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 shutdown_connections 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 --2- 172.21.15.100:0/2613200926 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fba74078ca0 0x7fba74112f50 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 >> 172.21.15.100:0/2613200926 conn(0x7fba740fde40 msgr2=0x7fba74109d70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 shutdown_connections 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:44.354+0000 7fba7bc47700 1 -- 172.21.15.100:0/2613200926 wait complete. 2024-03-24T11:09:44.951 INFO:teuthology.orchestra.run.smithi100.stdout:Generating new minimal ceph.conf... 2024-03-24T11:09:46.010 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:45 smithi100 ceph-mon[28571]: from='client.? 172.21.15.100:0/2617594311' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1cdd48700 1 Processor -- start 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1cdd48700 1 -- start start 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1cdd48700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8077200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1cdd48700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c80790b0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1c77fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8077200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1c77fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8077200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50944/0 (socket says 172.21.15.100:50944) 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1c77fe700 1 -- 172.21.15.100:0/3442010314 learned_addr learned my addr 172.21.15.100:0/3442010314 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:46.017 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1c77fe700 1 -- 172.21.15.100:0/3442010314 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1c8077740 con 0x7fa1c8078ca0 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1c77fe700 1 --2- 172.21.15.100:0/3442010314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8077200 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0098e0 tx=0x7fa1bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=be76fcfff1def28f server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.353+0000 7fa1c67fc700 1 -- 172.21.15.100:0/3442010314 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1bc00b860 con 0x7fa1c8078ca0 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1c67fc700 1 -- 172.21.15.100:0/3442010314 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa1bc00be50 con 0x7fa1c8078ca0 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1c67fc700 1 -- 172.21.15.100:0/3442010314 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1bc010380 con 0x7fa1c8078ca0 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 -- 172.21.15.100:0/3442010314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 msgr2=0x7fa1c8077200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 --2- 172.21.15.100:0/3442010314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8077200 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc0098e0 tx=0x7fa1bc009250 comp rx=0 tx=0).stop 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 -- 172.21.15.100:0/3442010314 shutdown_connections 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 --2- 172.21.15.100:0/3442010314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8077200 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 -- 172.21.15.100:0/3442010314 >> 172.21.15.100:0/3442010314 conn(0x7fa1c80fdea0 msgr2=0x7fa1c81002b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 -- 172.21.15.100:0/3442010314 shutdown_connections 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.354+0000 7fa1cdd48700 1 -- 172.21.15.100:0/3442010314 wait complete. 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1cdd48700 1 Processor -- start 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1cdd48700 1 -- start start 2024-03-24T11:09:46.018 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1cdd48700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8113800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1cdd48700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1c8113d40 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1c77fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8113800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1c77fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8113800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50946/0 (socket says 172.21.15.100:50946) 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1c77fe700 1 -- 172.21.15.100:0/2617594311 learned_addr learned my addr 172.21.15.100:0/2617594311 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1c77fe700 1 -- 172.21.15.100:0/2617594311 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1bc0094f0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.355+0000 7fa1c77fe700 1 --2- 172.21.15.100:0/2617594311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8113800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00b440 tx=0x7fa1bc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1bc010760 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fa1bc010d50 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1c8113f40 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1bc01adc0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1c8114ab0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7fa1bc024670 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.356+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fa1bc019880 con 0x7fa1c8078ca0 2024-03-24T11:09:46.019 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.357+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1c805f5f0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.359+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7fa1bc029060 con 0x7fa1c8078ca0 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.424+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa1c81089c0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.424+0000 7fa1affff700 1 -- 172.21.15.100:0/2617594311 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fa1bc010ec0 con 0x7fa1c8078ca0 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 msgr2=0x7fa1c8113800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 --2- 172.21.15.100:0/2617594311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8113800 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fa1bc00b440 tx=0x7fa1bc009250 comp rx=0 tx=0).stop 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 shutdown_connections 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 --2- 172.21.15.100:0/2617594311 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa1c8078ca0 0x7fa1c8113800 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 >> 172.21.15.100:0/2617594311 conn(0x7fa1c80fdea0 msgr2=0x7fa1c80feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 shutdown_connections 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:45.425+0000 7fa1cdd48700 1 -- 172.21.15.100:0/2617594311 wait complete. 2024-03-24T11:09:46.020 INFO:teuthology.orchestra.run.smithi100.stdout:Restarting the monitor... 2024-03-24T11:09:46.280 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:46 smithi100 systemd[1]: Stopping Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:09:46.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28567]: 2024-03-24T11:09:46.279+0000 7f1e55039700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi100 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:09:46.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28567]: 2024-03-24T11:09:46.279+0000 7f1e55039700 -1 mon.smithi100@0(leader) e1 *** Got Signal Terminated *** 2024-03-24T11:09:46.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:46 smithi100 podman[28816]: 2024-03-24 11:09:46.525539343 +0000 UTC m=+0.354851417 container died 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-03-24T11:09:47.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:46 smithi100 podman[28816]: 2024-03-24 11:09:46.882257183 +0000 UTC m=+0.711569265 container cleanup 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, ceph=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS) 2024-03-24T11:09:47.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:46 smithi100 bash[28816]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100 2024-03-24T11:09:47.522 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 podman[28830]: 2024-03-24 11:09:47.16561566 +0000 UTC m=+0.636500756 container remove 5bd82a5610a8a241aac54bebbfd2ce2283ac1640fef0a5cf57e430a58c33496b (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, RELEASE=quincy-9c4e0c3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:09:47.522 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service: Deactivated successfully. 2024-03-24T11:09:47.522 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 systemd[1]: Stopped Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:09:47.522 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 systemd[1]: Starting Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 podman[28922]: 2024-03-24 11:09:47.522777158 +0000 UTC m=+0.113636649 container create f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3, CEPH_POINT_RELEASE=) 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 podman[28922]: 2024-03-24 11:09:47.426813793 +0000 UTC m=+0.017673286 image pull 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de quay.ceph.io/ceph-ci/ceph:quincy 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 podman[28922]: 2024-03-24 11:09:47.733632503 +0000 UTC m=+0.324492009 container init f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True) 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 podman[28922]: 2024-03-24 11:09:47.736441473 +0000 UTC m=+0.327300966 container start f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, ceph=True, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: set uid:gid to 167:167 (ceph:ceph) 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable), process ceph-mon, pid 2 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: pidfile_write: ignore empty --pid-file 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: load: jerasure load: lrc 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: RocksDB version: 6.15.5 2024-03-24T11:09:47.817 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Compile date Mar 21 2024 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: DB SUMMARY 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: DB Session ID: Q196ANEEVVY1K1JI9F3E 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: CURRENT file: CURRENT 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: IDENTITY file: IDENTITY 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi100/store.db dir, Total Num: 1, files: 000008.sst 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi100/store.db: 000010.log size: 97306 ; 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.error_if_exists: 0 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.create_if_missing: 0 2024-03-24T11:09:47.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.paranoid_checks: 1 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.env: 0x5589d6995a80 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.fs: Posix File System 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.info_log: 0x5589d74e9c00 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_file_opening_threads: 16 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.statistics: (nil) 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.use_fsync: 0 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_log_file_size: 0 2024-03-24T11:09:47.820 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.log_file_time_to_roll: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.keep_log_file_num: 1000 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.recycle_log_file_num: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.allow_fallocate: 1 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.allow_mmap_reads: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.allow_mmap_writes: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.use_direct_reads: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.create_missing_column_families: 0 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.db_log_dir: 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi100/store.db 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.table_cache_numshardbits: 6 2024-03-24T11:09:47.821 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.WAL_ttl_seconds: 0 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.WAL_size_limit_MB: 0 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.is_fd_close_on_exec: 1 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.advise_random_on_open: 1 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.db_write_buffer_size: 0 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.write_buffer_manager: 0x5589d825ea50 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.use_adaptive_mutex: 0 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.rate_limiter: (nil) 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-03-24T11:09:47.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.wal_recovery_mode: 2 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.enable_thread_tracking: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.enable_pipelined_write: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.unordered_write: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.row_cache: None 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.wal_filter: None 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.allow_ingest_behind: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.preserve_deletes: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.two_write_queues: 0 2024-03-24T11:09:47.823 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.manual_wal_flush: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.atomic_flush: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.persist_stats_to_disk: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.write_dbid_to_manifest: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.log_readahead_size: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.best_efforts_recovery: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.allow_data_in_errors: 0 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.db_host_id: __hostname__ 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_background_jobs: 2 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_background_compactions: -1 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_subcompactions: 1 2024-03-24T11:09:47.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.delayed_write_rate : 16777216 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_total_wal_size: 0 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.stats_dump_period_sec: 600 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.stats_persist_period_sec: 600 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_open_files: -1 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bytes_per_sync: 0 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.wal_bytes_per_sync: 0 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.strict_bytes_per_sync: 0 2024-03-24T11:09:47.825 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_readahead_size: 0 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_background_flushes: -1 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Compression algorithms supported: 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kZSTD supported: 0 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kXpressCompression supported: 0 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kLZ4HCCompression supported: 1 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kZlibCompression supported: 1 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kSnappyCompression supported: 1 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kLZ4Compression supported: 1 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: kBZip2Compression supported: 0 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Fast CRC32 supported: Supported on x86 2024-03-24T11:09:47.826 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi100/store.db/MANIFEST-000009 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.merge_operator: 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_filter: None 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_filter_factory: None 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.sst_partitioner_factory: None 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.memtable_factory: SkipListFactory 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.table_factory: BlockBasedTable 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5589d753ef58) 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: cache_index_and_filter_blocks: 1 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: pin_top_level_index_and_filter: 1 2024-03-24T11:09:47.827 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: index_type: 0 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: data_block_index_type: 0 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: index_shortening: 1 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: data_block_hash_table_util_ratio: 0.750000 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: hash_index_allow_collision: 1 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: checksum: 1 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: no_block_cache: 0 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_cache: 0x5589d7571090 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_cache_name: BinnedLRUCache 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_cache_options: 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: capacity : 536870912 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: num_shard_bits : 4 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: strict_capacity_limit : 0 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: high_pri_pool_ratio: 0.000 2024-03-24T11:09:47.828 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_cache_compressed: (nil) 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: persistent_cache: (nil) 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_size: 4096 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_size_deviation: 10 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_restart_interval: 16 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: index_block_restart_interval: 1 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: metadata_block_size: 4096 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: partition_filters: 0 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: use_delta_encoding: 1 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: whole_key_filtering: 1 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: verify_compression: 0 2024-03-24T11:09:47.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: read_amp_bytes_per_bit: 0 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: format_version: 4 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: enable_index_compression: 1 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: block_align: 0 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.write_buffer_size: 33554432 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_write_buffer_number: 2 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression: NoCompression 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression: Disabled 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.prefix_extractor: nullptr 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.num_levels: 7 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-03-24T11:09:47.830 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.window_bits: -14 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.level: 32767 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.strategy: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compression_opts.enabled: false 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-03-24T11:09:47.831 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.target_file_size_base: 67108864 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.target_file_size_multiplier: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-03-24T11:09:47.832 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.arena_block_size: 4194304 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.disable_auto_compactions: 0 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-03-24T11:09:47.833 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.table_properties_collectors: 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.inplace_update_support: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.inplace_update_num_locks: 10000 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.memtable_huge_page_size: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.bloom_locality: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.max_successive_merges: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.optimize_filters_for_hits: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.paranoid_file_checks: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.force_consistency_checks: 1 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.report_bg_io_stats: 0 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.ttl: 2592000 2024-03-24T11:09:47.834 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.periodic_compaction_seconds: 0 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.enable_blob_files: false 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.min_blob_size: 0 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.blob_file_size: 268435456 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.blob_compression_type: NoCompression 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.enable_blob_garbage_collection: false 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi100/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278587761963, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-03-24T11:09:47.835 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278587762791, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92578, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90673, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13356, "raw_average_key_size": 49, "raw_value_size": 83388, "raw_average_value_size": 309, "num_data_blocks": 11, "num_entries": 269, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1711278587, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "df1d9f1e-9b85-4366-8af6-110161f50f10", "db_session_id": "Q196ANEEVVY1K1JI9F3E"}} 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: EVENT_LOG_v1 {"time_micros": 1711278587763117, "job": 1, "event": "recovery_finished"} 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi100/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x5589d7505180 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: DB pointer 0x5589d75da000 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** DB Stats ** 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-03-24T11:09:47.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** Compaction Stats [default] ** 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: L0 2/0 92.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Sum 2/0 92.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** Compaction Stats [default] ** 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.837 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Cumulative compaction: 0.00 GB write, 29.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Interval compaction: 0.00 GB write, 29.95 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** Compaction Stats [default] ** 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:09:47.838 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: L0 2/0 92.21 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Sum 2/0 92.21 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** Compaction Stats [default] ** 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 151.2 0.00 0.00 1 0.001 0 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: AddFile(Keys): cumulative 0, interval 0 2024-03-24T11:09:47.839 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Cumulative compaction: 0.00 GB write, 29.66 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: ** File Read Latency Histogram By Level [default] ** 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: starting mon.smithi100 rank 0 at public addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] at bind addrs [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi100 fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???) e1 preinit fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).mds e1 new map 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).mds e1 print_map 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: e1 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: legacy client fscid: -1 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: 2024-03-24T11:09:47.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout: No filesystems configured 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 bash[28922]: f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 2024-03-24T11:09:47.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 systemd[1]: Started Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:09:47.841 INFO:teuthology.orchestra.run.smithi100.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-03-24T11:09:48.074 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mon.smithi100 is new leader, mons smithi100 in quorum (ranks 0) 2024-03-24T11:09:48.075 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: monmap e1: 1 mons at {smithi100=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:09:48.075 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: fsmap 2024-03-24T11:09:48.075 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: osdmap e1: 0 total, 0 up, 0 in 2024-03-24T11:09:48.075 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:47 smithi100 ceph-mon[28941]: mgrmap e1: no daemons active 2024-03-24T11:09:49.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:49 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2539348708' entity='client.admin' 2024-03-24T11:09:53.704 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf92c6700 1 Processor -- start 2024-03-24T11:09:53.704 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf92c6700 1 -- start start 2024-03-24T11:09:53.704 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf92c6700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4101a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:53.704 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf92c6700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4101f70 con 0x7f1cf4101620 2024-03-24T11:09:53.704 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:53.704 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4101a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56940/0 (socket says 172.21.15.100:56940) 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.258+0000 7f1cf2ffd700 1 -- 172.21.15.100:0/1241382341 learned_addr learned my addr 172.21.15.100:0/1241382341 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.259+0000 7f1cf2ffd700 1 -- 172.21.15.100:0/1241382341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf41020b0 con 0x7f1cf4101620 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.259+0000 7f1cf2ffd700 1 --2- 172.21.15.100:0/1241382341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4101a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40098e0 tx=0x7f1ce4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f86c67c448735d42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.259+0000 7f1cf1ffb700 1 -- 172.21.15.100:0/1241382341 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce400b860 con 0x7f1cf4101620 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.259+0000 7f1cf1ffb700 1 -- 172.21.15.100:0/1241382341 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1ce400be50 con 0x7f1cf4101620 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.259+0000 7f1cf1ffb700 1 -- 172.21.15.100:0/1241382341 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce4010380 con 0x7f1cf4101620 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- 172.21.15.100:0/1241382341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 msgr2=0x7f1cf4101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 --2- 172.21.15.100:0/1241382341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4101a30 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1ce40098e0 tx=0x7f1ce4009250 comp rx=0 tx=0).stop 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- 172.21.15.100:0/1241382341 shutdown_connections 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 --2- 172.21.15.100:0/1241382341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4101a30 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- 172.21.15.100:0/1241382341 >> 172.21.15.100:0/1241382341 conn(0x7f1cf40fcdc0 msgr2=0x7f1cf40ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- 172.21.15.100:0/1241382341 shutdown_connections 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- 172.21.15.100:0/1241382341 wait complete. 2024-03-24T11:09:53.705 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 Processor -- start 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- start start 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4076ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.260+0000 7f1cf92c6700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cf4112070 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4076ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4076ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56946/0 (socket says 172.21.15.100:56946) 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf2ffd700 1 -- 172.21.15.100:0/2539348708 learned_addr learned my addr 172.21.15.100:0/2539348708 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf2ffd700 1 -- 172.21.15.100:0/2539348708 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ce40094f0 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf2ffd700 1 --2- 172.21.15.100:0/2539348708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4076ab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400b440 tx=0x7f1ce4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce4010760 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1ce4010d50 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ce401ae20 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cf4116680 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cf4116a50 con 0x7f1cf4101620 2024-03-24T11:09:53.706 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.261+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1ce402e6c0 con 0x7f1cf4101620 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.262+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1ce4024910 con 0x7f1cf4101620 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.262+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cf404bed0 con 0x7f1cf4101620 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.264+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1ce4029070 con 0x7f1cf4101620 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.330+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f1cf4103440 con 0x7f1cf4101620 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cdb7fe700 1 -- 172.21.15.100:0/2539348708 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1cf4103440 con 0x7f1cf4101620 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 msgr2=0x7f1cf4076ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 --2- 172.21.15.100:0/2539348708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4076ab0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f1ce400b440 tx=0x7f1ce4009250 comp rx=0 tx=0).stop 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 shutdown_connections 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 --2- 172.21.15.100:0/2539348708 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1cf4101620 0x7f1cf4076ab0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 >> 172.21.15.100:0/2539348708 conn(0x7f1cf40fcdc0 msgr2=0x7f1cf40fd940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 shutdown_connections 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:48.335+0000 7f1cf92c6700 1 -- 172.21.15.100:0/2539348708 wait complete. 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:Wrote config to /etc/ceph/ceph.conf 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:09:53.707 INFO:teuthology.orchestra.run.smithi100.stdout:Creating mgr... 2024-03-24T11:09:53.708 INFO:teuthology.orchestra.run.smithi100.stdout:Verifying port 9283 ... 2024-03-24T11:09:53.916 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mgr.smithi100.smxgyi 2024-03-24T11:09:53.916 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Failed to reset failed state of unit ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mgr.smithi100.smxgyi.service: Unit ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mgr.smithi100.smxgyi.service not loaded. 2024-03-24T11:09:54.149 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715.target.wants/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mgr.smithi100.smxgyi.service → /etc/systemd/system/ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@.service. 2024-03-24T11:09:57.609 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:09:57.609 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-03-24T11:09:57.616 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:09:57.616 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-03-24T11:09:57.616 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-03-24T11:09:57.616 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to enable service . firewalld.service is not available 2024-03-24T11:09:57.624 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:09:57.624 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-03-24T11:09:57.632 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:09:57.632 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-03-24T11:09:57.632 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-03-24T11:09:57.632 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-03-24T11:09:57.632 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr to start... 2024-03-24T11:09:57.632 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr... 2024-03-24T11:09:58.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:09:58 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/966641737' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:09:58.815 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:09:58.815 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:09:58.815 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e2f9ef6c-e9ce-11ee-95cd-87774f69a715", 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:09:58.816 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "smithi100" 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 10, 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:09:58.817 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:09:58.825 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:09:58.826 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:09:42.711437+0000", 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:09:58.827 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.111+0000 7f1dfe428700 1 Processor -- start 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.111+0000 7f1dfe428700 1 -- start start 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.111+0000 7f1dfe428700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.111+0000 7f1dfe428700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df81058f0 con 0x7f1df8076bb0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40362/0 (socket says 172.21.15.100:40362) 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df7fff700 1 -- 172.21.15.100:0/3249126439 learned_addr learned my addr 172.21.15.100:0/3249126439 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df7fff700 1 -- 172.21.15.100:0/3249126439 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df8105a30 con 0x7f1df8076bb0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df7fff700 1 --2- 172.21.15.100:0/3249126439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df81053b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098e0 tx=0x7f1de8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=43df5bc26a86184b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df6ffd700 1 -- 172.21.15.100:0/3249126439 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de800b860 con 0x7f1df8076bb0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df6ffd700 1 -- 172.21.15.100:0/3249126439 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1de800be50 con 0x7f1df8076bb0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.112+0000 7f1df6ffd700 1 -- 172.21.15.100:0/3249126439 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de8010380 con 0x7f1df8076bb0 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.113+0000 7f1dfe428700 1 -- 172.21.15.100:0/3249126439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 msgr2=0x7f1df81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:58.828 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.113+0000 7f1dfe428700 1 --2- 172.21.15.100:0/3249126439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df81053b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7f1de80098e0 tx=0x7f1de8009250 comp rx=0 tx=0).stop 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.113+0000 7f1dfe428700 1 -- 172.21.15.100:0/3249126439 shutdown_connections 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.113+0000 7f1dfe428700 1 --2- 172.21.15.100:0/3249126439 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df81053b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.113+0000 7f1dfe428700 1 -- 172.21.15.100:0/3249126439 >> 172.21.15.100:0/3249126439 conn(0x7f1df80fce40 msgr2=0x7f1df80ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.113+0000 7f1dfe428700 1 -- 172.21.15.100:0/3249126439 shutdown_connections 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.114+0000 7f1dfe428700 1 -- 172.21.15.100:0/3249126439 wait complete. 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.114+0000 7f1dfe428700 1 Processor -- start 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.114+0000 7f1dfe428700 1 -- start start 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.114+0000 7f1dfe428700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df8102d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.114+0000 7f1dfe428700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1df8103240 con 0x7f1df8076bb0 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df8102d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df8102d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40374/0 (socket says 172.21.15.100:40374) 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df7fff700 1 -- 172.21.15.100:0/966641737 learned_addr learned my addr 172.21.15.100:0/966641737 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df7fff700 1 -- 172.21.15.100:0/966641737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1de80094f0 con 0x7f1df8076bb0 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df7fff700 1 --2- 172.21.15.100:0/966641737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df8102d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1de8011bd0 tx=0x7f1de8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:09:58.829 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de80107c0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f1de8010db0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1df81012b0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1df81017a0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.115+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1de801ad60 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.116+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f1de802e6c0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.116+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1de80248d0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.117+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1df804bed0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.119+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f1de801a5c0 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.190+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f1df8101c20 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.190+0000 7f1df4ff9700 1 -- 172.21.15.100:0/966641737 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f1de8010920 con 0x7f1df8076bb0 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.191+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 msgr2=0x7f1df8102d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.191+0000 7f1dfe428700 1 --2- 172.21.15.100:0/966641737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df8102d00 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f1de8011bd0 tx=0x7f1de8009250 comp rx=0 tx=0).stop 2024-03-24T11:09:58.830 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.191+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 shutdown_connections 2024-03-24T11:09:58.831 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.191+0000 7f1dfe428700 1 --2- 172.21.15.100:0/966641737 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1df8076bb0 0x7f1df8102d00 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:09:58.831 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.191+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 >> 172.21.15.100:0/966641737 conn(0x7f1df80fce40 msgr2=0x7f1df80fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:09:58.831 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.192+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 shutdown_connections 2024-03-24T11:09:58.831 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:09:58.192+0000 7f1dfe428700 1 -- 172.21.15.100:0/966641737 wait complete. 2024-03-24T11:09:58.831 INFO:teuthology.orchestra.run.smithi100.stdout:mgr not available, waiting (1/15)... 2024-03-24T11:10:01.762 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:01 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1719557240' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:10:02.102 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:10:02.104 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:02.104 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e2f9ef6c-e9ce-11ee-95cd-87774f69a715", 2024-03-24T11:10:02.104 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:10:02.104 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "smithi100" 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 13, 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:02.105 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:10:02.106 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:10:02.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:10:02.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:10:02.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:10:02.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:10:02.115 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:09:42.711437+0000", 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:10:02.116 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.458+0000 7f70ccf29700 1 Processor -- start 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.458+0000 7f70ccf29700 1 -- start start 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.459+0000 7f70ccf29700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8101a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.459+0000 7f70ccf29700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c8101f50 con 0x7f70c8101600 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.459+0000 7f70c659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8101a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.459+0000 7f70c659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8101a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40386/0 (socket says 172.21.15.100:40386) 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.459+0000 7f70c659c700 1 -- 172.21.15.100:0/1542179031 learned_addr learned my addr 172.21.15.100:0/1542179031 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.459+0000 7f70c659c700 1 -- 172.21.15.100:0/1542179031 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70c8102090 con 0x7f70c8101600 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.460+0000 7f70c659c700 1 --2- 172.21.15.100:0/1542179031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8101a10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70b80098e0 tx=0x7f70b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a3a46fa572acab63 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.460+0000 7f70c559a700 1 -- 172.21.15.100:0/1542179031 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b800b860 con 0x7f70c8101600 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.460+0000 7f70c559a700 1 -- 172.21.15.100:0/1542179031 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f70b800be50 con 0x7f70c8101600 2024-03-24T11:10:02.117 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.460+0000 7f70c559a700 1 -- 172.21.15.100:0/1542179031 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8010380 con 0x7f70c8101600 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.460+0000 7f70ccf29700 1 -- 172.21.15.100:0/1542179031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 msgr2=0x7f70c8101a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.460+0000 7f70ccf29700 1 --2- 172.21.15.100:0/1542179031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8101a10 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f70b80098e0 tx=0x7f70b8009250 comp rx=0 tx=0).stop 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 -- 172.21.15.100:0/1542179031 shutdown_connections 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 --2- 172.21.15.100:0/1542179031 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8101a10 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 -- 172.21.15.100:0/1542179031 >> 172.21.15.100:0/1542179031 conn(0x7f70c80fcd80 msgr2=0x7f70c80ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 -- 172.21.15.100:0/1542179031 shutdown_connections 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 -- 172.21.15.100:0/1542179031 wait complete. 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 Processor -- start 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 -- start start 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8076b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70ccf29700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70c81141e0 con 0x7f70c8101600 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70c659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8076b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70c659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8076b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40400/0 (socket says 172.21.15.100:40400) 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.461+0000 7f70c659c700 1 -- 172.21.15.100:0/1719557240 learned_addr learned my addr 172.21.15.100:0/1719557240 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:02.118 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70c659c700 1 -- 172.21.15.100:0/1719557240 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70b80094f0 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70c659c700 1 --2- 172.21.15.100:0/1719557240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8076b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b440 tx=0x7f70b8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b8010760 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70c8114320 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70c8114810 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f70b8010d50 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.462+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70b801ae20 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.463+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 1) v1 ==== 521+0+0 (secure 0 0 0) 0x7f70b80246f0 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.463+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f70b802e9c0 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.463+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70a80052e0 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.467+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f70b8029070 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.548+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f70a8005860 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.549+0000 7f70b6ffd700 1 -- 172.21.15.100:0/1719557240 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f70b802ebd0 con 0x7f70c8101600 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 msgr2=0x7f70c8076b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:02.119 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 --2- 172.21.15.100:0/1719557240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8076b10 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f70b800b440 tx=0x7f70b8009250 comp rx=0 tx=0).stop 2024-03-24T11:10:02.120 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 shutdown_connections 2024-03-24T11:10:02.120 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 --2- 172.21.15.100:0/1719557240 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70c8101600 0x7f70c8076b10 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:02.120 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 >> 172.21.15.100:0/1719557240 conn(0x7f70c80fcd80 msgr2=0x7f70c80fd8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:02.120 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 shutdown_connections 2024-03-24T11:10:02.120 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:01.550+0000 7f70ccf29700 1 -- 172.21.15.100:0/1719557240 wait complete. 2024-03-24T11:10:02.120 INFO:teuthology.orchestra.run.smithi100.stdout:mgr not available, waiting (2/15)... 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: Activating manager daemon smithi100.smxgyi 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: mgrmap e2: smithi100.smxgyi(active, starting, since 0.00304763s) 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi100.smxgyi", "id": "smithi100.smxgyi"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: Manager daemon smithi100.smxgyi is now available 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/trash_purge_schedule"}]: dispatch 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:04.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:04.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:03 smithi100 ceph-mon[28941]: from='mgr.14100 172.21.15.100:0/2882676570' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:05.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:04 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/227403699' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:10:05.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:04 smithi100 ceph-mon[28941]: mgrmap e3: smithi100.smxgyi(active, since 1.00551s) 2024-03-24T11:10:05.214 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e2f9ef6c-e9ce-11ee-95cd-87774f69a715", 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:10:05.215 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "smithi100" 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 16, 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:10:05.216 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.217 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:10:05.217 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:05.217 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:10:05.217 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:10:05.217 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:10:05.217 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:10:05.218 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:10:05.218 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:10:05.218 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.218 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:10:05.218 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:10:05.218 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:10:05.219 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:10:05.220 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": false, 2024-03-24T11:10:05.221 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:10:05.221 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:10:05.221 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:10:05.221 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:10:05.221 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:10:05.221 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:09:42.711437+0000", 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:10:05.222 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:05.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:10:05.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:05.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.525+0000 7f76c4884700 1 Processor -- start 2024-03-24T11:10:05.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.525+0000 7f76c4884700 1 -- start start 2024-03-24T11:10:05.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.525+0000 7f76c4884700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c01052e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:05.223 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.525+0000 7f76c4884700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0105820 con 0x7f76c0076be0 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76be59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c01052e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76be59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c01052e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40474/0 (socket says 172.21.15.100:40474) 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76be59c700 1 -- 172.21.15.100:0/3628938697 learned_addr learned my addr 172.21.15.100:0/3628938697 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76be59c700 1 -- 172.21.15.100:0/3628938697 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76c0105960 con 0x7f76c0076be0 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76be59c700 1 --2- 172.21.15.100:0/3628938697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c01052e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f76b00098e0 tx=0x7f76b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83eb1c1f715c9277 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76bd59a700 1 -- 172.21.15.100:0/3628938697 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b000b860 con 0x7f76c0076be0 2024-03-24T11:10:05.224 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76bd59a700 1 -- 172.21.15.100:0/3628938697 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f76b000be50 con 0x7f76c0076be0 2024-03-24T11:10:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.526+0000 7f76bd59a700 1 -- 172.21.15.100:0/3628938697 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b0010380 con 0x7f76c0076be0 2024-03-24T11:10:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 -- 172.21.15.100:0/3628938697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 msgr2=0x7f76c01052e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 --2- 172.21.15.100:0/3628938697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c01052e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f76b00098e0 tx=0x7f76b0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 -- 172.21.15.100:0/3628938697 shutdown_connections 2024-03-24T11:10:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 --2- 172.21.15.100:0/3628938697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c01052e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:05.225 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 -- 172.21.15.100:0/3628938697 >> 172.21.15.100:0/3628938697 conn(0x7f76c00fcd50 msgr2=0x7f76c00ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 -- 172.21.15.100:0/3628938697 shutdown_connections 2024-03-24T11:10:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 -- 172.21.15.100:0/3628938697 wait complete. 2024-03-24T11:10:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.527+0000 7f76c4884700 1 Processor -- start 2024-03-24T11:10:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76c4884700 1 -- start start 2024-03-24T11:10:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76c4884700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c0111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:05.226 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76c4884700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76c0111aa0 con 0x7f76c0076be0 2024-03-24T11:10:05.227 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76be59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c0111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:05.227 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76be59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c0111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40486/0 (socket says 172.21.15.100:40486) 2024-03-24T11:10:05.227 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76be59c700 1 -- 172.21.15.100:0/227403699 learned_addr learned my addr 172.21.15.100:0/227403699 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:05.227 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76be59c700 1 -- 172.21.15.100:0/227403699 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76b00094f0 con 0x7f76c0076be0 2024-03-24T11:10:05.227 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76be59c700 1 --2- 172.21.15.100:0/227403699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c0111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f76b0003050 tx=0x7f76b0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:05.227 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b0010760 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f76b0010d50 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76c0111ca0 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76b001ae20 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.528+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76c0112810 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.529+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 2) v1 ==== 43124+0+0 (secure 0 0 0) 0x7f76b00108c0 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.529+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f76b004a020 con 0x7f76c0076be0 2024-03-24T11:10:05.228 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.529+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76c005f5f0 con 0x7f76c0076be0 2024-03-24T11:10:05.229 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.532+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71349 (secure 0 0 0) 0x7f76b001a6a0 con 0x7f76c0076be0 2024-03-24T11:10:05.229 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.599+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f76c0112a50 con 0x7f76c0076be0 2024-03-24T11:10:05.229 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76aeffd700 1 -- 172.21.15.100:0/227403699 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f76b0032500 con 0x7f76c0076be0 2024-03-24T11:10:05.229 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 msgr2=0x7f76c0111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:05.229 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 --2- 172.21.15.100:0/227403699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c0111560 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f76b0003050 tx=0x7f76b0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:05.229 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 shutdown_connections 2024-03-24T11:10:05.230 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 --2- 172.21.15.100:0/227403699 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f76c0076be0 0x7f76c0111560 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:05.230 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 >> 172.21.15.100:0/227403699 conn(0x7f76c00fcd50 msgr2=0x7f76c00fd950 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:05.230 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 shutdown_connections 2024-03-24T11:10:05.230 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:04.600+0000 7f76c4884700 1 -- 172.21.15.100:0/227403699 wait complete. 2024-03-24T11:10:05.230 INFO:teuthology.orchestra.run.smithi100.stdout:mgr not available, waiting (3/15)... 2024-03-24T11:10:07.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:06 smithi100 ceph-mon[28941]: mgrmap e4: smithi100.smxgyi(active, since 2s) 2024-03-24T11:10:08.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/764352797' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsid": "e2f9ef6c-e9ce-11ee-95cd-87774f69a715", 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "health": { 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 0 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-03-24T11:10:08.604 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "smithi100" 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "quorum_age": 20, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "monmap": { 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-03-24T11:10:08.605 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-03-24T11:10:08.606 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modules": [ 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "iostat", 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "nfs", 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "restful" 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ], 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-03-24T11:10:08.607 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "modified": "2024-03-24T11:09:42.711437+0000", 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "services": {} 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout }, 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d3df2d700 1 Processor -- start 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d3df2d700 1 -- start start 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d3df2d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d3df2d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d38103070 con 0x7f3d38102720 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d377fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d377fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48206/0 (socket says 172.21.15.100:48206) 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.612+0000 7f3d377fe700 1 -- 172.21.15.100:0/3310509953 learned_addr learned my addr 172.21.15.100:0/3310509953 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d377fe700 1 -- 172.21.15.100:0/3310509953 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d381031b0 con 0x7f3d38102720 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d377fe700 1 --2- 172.21.15.100:0/3310509953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38102b30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098e0 tx=0x7f3d28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ea7e6aca7cd7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:08.608 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d36ffd700 1 -- 172.21.15.100:0/3310509953 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2800b860 con 0x7f3d38102720 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d36ffd700 1 -- 172.21.15.100:0/3310509953 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3d2800be50 con 0x7f3d38102720 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d36ffd700 1 -- 172.21.15.100:0/3310509953 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d28010380 con 0x7f3d38102720 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d3df2d700 1 -- 172.21.15.100:0/3310509953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 msgr2=0x7f3d38102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.613+0000 7f3d3df2d700 1 --2- 172.21.15.100:0/3310509953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38102b30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3d280098e0 tx=0x7f3d28009250 comp rx=0 tx=0).stop 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 -- 172.21.15.100:0/3310509953 shutdown_connections 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 --2- 172.21.15.100:0/3310509953 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38102b30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 -- 172.21.15.100:0/3310509953 >> 172.21.15.100:0/3310509953 conn(0x7f3d380fdee0 msgr2=0x7f3d381002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 -- 172.21.15.100:0/3310509953 shutdown_connections 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 -- 172.21.15.100:0/3310509953 wait complete. 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 Processor -- start 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 -- start start 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38111560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.614+0000 7f3d3df2d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d38111aa0 con 0x7f3d38102720 2024-03-24T11:10:08.609 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d377fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38111560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d377fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38111560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48210/0 (socket says 172.21.15.100:48210) 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d377fe700 1 -- 172.21.15.100:0/764352797 learned_addr learned my addr 172.21.15.100:0/764352797 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d377fe700 1 -- 172.21.15.100:0/764352797 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d280094f0 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d377fe700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38111560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800b440 tx=0x7f3d28009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d28010760 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f3d28010d50 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d38111ca0 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d2801ae20 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.615+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d38112810 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.616+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f3d2802e440 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.616+0000 7f3d34ff9700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f3d24036790 0x7f3d24038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.616+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3d2804a660 con 0x7f3d38102720 2024-03-24T11:10:08.610 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.616+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d3805f5f0 con 0x7f3d38102720 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.616+0000 7f3d2effd700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f3d24036790 0x7f3d24038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.617+0000 7f3d2effd700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f3d24036790 0x7f3d24038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009990 tx=0x7f3d20006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.621+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3d28018350 con 0x7f3d38102720 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.885+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f3d38112b10 con 0x7f3d38102720 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.886+0000 7f3d34ff9700 1 -- 172.21.15.100:0/764352797 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7f3d28018350 con 0x7f3d38102720 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f3d24036790 msgr2=0x7f3d24038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f3d24036790 0x7f3d24038c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f3d20009990 tx=0x7f3d20006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 msgr2=0x7f3d38111560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38111560 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f3d2800b440 tx=0x7f3d28009250 comp rx=0 tx=0).stop 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 shutdown_connections 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f3d24036790 0x7f3d24038c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 --2- 172.21.15.100:0/764352797 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3d38102720 0x7f3d38111560 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 >> 172.21.15.100:0/764352797 conn(0x7f3d380fdee0 msgr2=0x7f3d380fea60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 shutdown_connections 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:07.889+0000 7f3d3df2d700 1 -- 172.21.15.100:0/764352797 wait complete. 2024-03-24T11:10:08.611 INFO:teuthology.orchestra.run.smithi100.stdout:mgr is available 2024-03-24T11:10:09.502 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:09 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/709833367' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [global] 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout fsid = e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-03-24T11:10:09.747 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [mgr] 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout [osd] 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfccc34700 1 Processor -- start 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfccc34700 1 -- start start 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfccc34700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:09.748 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfccc34700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc8103080 con 0x7fbfc8102730 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfc659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfc659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48214/0 (socket says 172.21.15.100:48214) 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.002+0000 7fbfc659c700 1 -- 172.21.15.100:0/2748052811 learned_addr learned my addr 172.21.15.100:0/2748052811 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.003+0000 7fbfc659c700 1 -- 172.21.15.100:0/2748052811 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc81031c0 con 0x7fbfc8102730 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.003+0000 7fbfc659c700 1 --2- 172.21.15.100:0/2748052811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8102b40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009e50 tx=0x7fbfc0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a83c00a0d5ad09a6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.003+0000 7fbfc559a700 1 -- 172.21.15.100:0/2748052811 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc00092e0 con 0x7fbfc8102730 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.003+0000 7fbfc559a700 1 -- 172.21.15.100:0/2748052811 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbfc0012430 con 0x7fbfc8102730 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.003+0000 7fbfc559a700 1 -- 172.21.15.100:0/2748052811 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc00075c0 con 0x7fbfc8102730 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- 172.21.15.100:0/2748052811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 msgr2=0x7fbfc8102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 --2- 172.21.15.100:0/2748052811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8102b40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009e50 tx=0x7fbfc0009840 comp rx=0 tx=0).stop 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- 172.21.15.100:0/2748052811 shutdown_connections 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 --2- 172.21.15.100:0/2748052811 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8102b40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- 172.21.15.100:0/2748052811 >> 172.21.15.100:0/2748052811 conn(0x7fbfc80fdf10 msgr2=0x7fbfc8100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:09.749 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- 172.21.15.100:0/2748052811 shutdown_connections 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- 172.21.15.100:0/2748052811 wait complete. 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 Processor -- start 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- start start 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8111630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfccc34700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfc8111b70 con 0x7fbfc8102730 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfc659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8111630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfc659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8111630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48222/0 (socket says 172.21.15.100:48222) 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.004+0000 7fbfc659c700 1 -- 172.21.15.100:0/709833367 learned_addr learned my addr 172.21.15.100:0/709833367 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfc659c700 1 -- 172.21.15.100:0/709833367 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfc0009ae0 con 0x7fbfc8102730 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfc659c700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8111630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009fd0 tx=0x7fbfc0009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc00092e0 con 0x7fbfc8102730 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fbfc0007d10 con 0x7fbfc8102730 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfc8111d70 con 0x7fbfc8102730 2024-03-24T11:10:09.750 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbfc001c8c0 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.005+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfc81128e0 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.006+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7fbfc0019070 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.006+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfc805f5f0 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.006+0000 7fbfb6ffd700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fbfb0036790 0x7fbfb0038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.006+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fbfc0049e60 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.006+0000 7fbfc5d9b700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fbfb0036790 0x7fbfb0038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.007+0000 7fbfc5d9b700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fbfb0036790 0x7fbfb0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009990 tx=0x7fbfbc006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.011+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbfc0019910 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.194+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fbfc8112c60 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.195+0000 7fbfb6ffd700 1 -- 172.21.15.100:0/709833367 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7fbfc0017020 con 0x7fbfc8102730 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fbfb0036790 msgr2=0x7fbfb0038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fbfb0036790 0x7fbfb0038c40 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fbfbc009990 tx=0x7fbfbc006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:09.751 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 msgr2=0x7fbfc8111630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8111630 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fbfc0009fd0 tx=0x7fbfc0009840 comp rx=0 tx=0).stop 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 shutdown_connections 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fbfb0036790 0x7fbfb0038c40 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 --2- 172.21.15.100:0/709833367 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbfc8102730 0x7fbfc8111630 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 >> 172.21.15.100:0/709833367 conn(0x7fbfc80fdf10 msgr2=0x7fbfc80feb10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 shutdown_connections 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:09.197+0000 7fbfccc34700 1 -- 172.21.15.100:0/709833367 wait complete. 2024-03-24T11:10:09.752 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling cephadm module... 2024-03-24T11:10:10.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:10 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3753602494' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ecee700 1 Processor -- start 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ecee700 1 -- start start 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ecee700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418105310 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ecee700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418105850 con 0x7f7418076be0 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ca8a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418105310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ca8a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418105310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48236/0 (socket says 172.21.15.100:48236) 2024-03-24T11:10:11.110 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ca8a700 1 -- 172.21.15.100:0/1227477242 learned_addr learned my addr 172.21.15.100:0/1227477242 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.189+0000 7f741ca8a700 1 -- 172.21.15.100:0/1227477242 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7418105990 con 0x7f7418076be0 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ca8a700 1 --2- 172.21.15.100:0/1227477242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418105310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f740c0098e0 tx=0x7f740c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=85c4c286da6d6322 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f74177fe700 1 -- 172.21.15.100:0/1227477242 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c00b860 con 0x7f7418076be0 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f74177fe700 1 -- 172.21.15.100:0/1227477242 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f740c00be50 con 0x7f7418076be0 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f74177fe700 1 -- 172.21.15.100:0/1227477242 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c010380 con 0x7f7418076be0 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 -- 172.21.15.100:0/1227477242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 msgr2=0x7f7418105310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 --2- 172.21.15.100:0/1227477242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418105310 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f740c0098e0 tx=0x7f740c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 -- 172.21.15.100:0/1227477242 shutdown_connections 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 --2- 172.21.15.100:0/1227477242 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418105310 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 -- 172.21.15.100:0/1227477242 >> 172.21.15.100:0/1227477242 conn(0x7f74180fcdc0 msgr2=0x7f74180ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 -- 172.21.15.100:0/1227477242 shutdown_connections 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.190+0000 7f741ecee700 1 -- 172.21.15.100:0/1227477242 wait complete. 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ecee700 1 Processor -- start 2024-03-24T11:10:11.111 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ecee700 1 -- start start 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ecee700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418111570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ecee700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7418111ab0 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ca8a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418111570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ca8a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418111570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48250/0 (socket says 172.21.15.100:48250) 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.191+0000 7f741ca8a700 1 -- 172.21.15.100:0/3753602494 learned_addr learned my addr 172.21.15.100:0/3753602494 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f741ca8a700 1 -- 172.21.15.100:0/3753602494 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f740c0094f0 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f741ca8a700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418111570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f740c003050 tx=0x7f740c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c010760 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f740c010d50 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7418111cb0 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f740c01ae20 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.192+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7418112820 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.193+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 4) v1 ==== 43357+0+0 (secure 0 0 0) 0x7f740c02e440 con 0x7f7418076be0 2024-03-24T11:10:11.112 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.193+0000 7f74157fa700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f7408036720 0x7f7408038bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.193+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f741805f5f0 con 0x7f7418076be0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.193+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f740c04a460 con 0x7f7418076be0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.193+0000 7f7417fff700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f7408036720 0x7f7408038bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.193+0000 7f7417fff700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f7408036720 0x7f7408038bd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7404009990 tx=0x7f7404006e40 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.198+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f740c0243f0 con 0x7f7418076be0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.433+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f741804bed0 con 0x7f7418076be0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.486+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f740c00ec20 con 0x7f7418076be0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.487+0000 7f74157fa700 1 -- 172.21.15.100:0/3753602494 <== mon.0 v2:172.21.15.100:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f740c018760 con 0x7f7418076be0 2024-03-24T11:10:11.113 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f7408036720 msgr2=0x7f7408038bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f7408036720 0x7f7408038bd0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f7404009990 tx=0x7f7404006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 msgr2=0x7f7418111570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418111570 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f740c003050 tx=0x7f740c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 shutdown_connections 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f7408036720 0x7f7408038bd0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 --2- 172.21.15.100:0/3753602494 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7418076be0 0x7f7418111570 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 >> 172.21.15.100:0/3753602494 conn(0x7f74180fcdc0 msgr2=0x7f74180fd960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 shutdown_connections 2024-03-24T11:10:11.114 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:10.489+0000 7f741ecee700 1 -- 172.21.15.100:0/3753602494 wait complete. 2024-03-24T11:10:11.715 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:11 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3753602494' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-03-24T11:10:11.715 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:11 smithi100 ceph-mon[28941]: mgrmap e5: smithi100.smxgyi(active, since 6s) 2024-03-24T11:10:12.384 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 5, 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "active_name": "smithi100.smxgyi", 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a91791700 1 Processor -- start 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a91791700 1 -- start start 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a91791700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c100f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a91791700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c101470 con 0x7f0a8c100b20 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a8affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c100f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a8affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c100f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48276/0 (socket says 172.21.15.100:48276) 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a8affd700 1 -- 172.21.15.100:0/1706108748 learned_addr learned my addr 172.21.15.100:0/1706108748 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:12.385 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.510+0000 7f0a8affd700 1 -- 172.21.15.100:0/1706108748 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a8c1015b0 con 0x7f0a8c100b20 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a8affd700 1 --2- 172.21.15.100:0/1706108748 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c100f30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0098e0 tx=0x7f0a7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7f2bd429303e5b30 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a89ffb700 1 -- 172.21.15.100:0/1706108748 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a7c00b860 con 0x7f0a8c100b20 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a89ffb700 1 -- 172.21.15.100:0/1706108748 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0a7c00be50 con 0x7f0a8c100b20 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a89ffb700 1 -- 172.21.15.100:0/1706108748 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a7c010380 con 0x7f0a8c100b20 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 -- 172.21.15.100:0/1706108748 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 msgr2=0x7f0a8c100f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 --2- 172.21.15.100:0/1706108748 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c100f30 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c0098e0 tx=0x7f0a7c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 -- 172.21.15.100:0/1706108748 shutdown_connections 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 --2- 172.21.15.100:0/1706108748 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c100f30 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 -- 172.21.15.100:0/1706108748 >> 172.21.15.100:0/1706108748 conn(0x7f0a8c0fc500 msgr2=0x7f0a8c0fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 -- 172.21.15.100:0/1706108748 shutdown_connections 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.511+0000 7f0a91791700 1 -- 172.21.15.100:0/1706108748 wait complete. 2024-03-24T11:10:12.386 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a91791700 1 Processor -- start 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a91791700 1 -- start start 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a91791700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c111540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a91791700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a8c111a80 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a8affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c111540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a8affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c111540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48280/0 (socket says 172.21.15.100:48280) 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.512+0000 7f0a8affd700 1 -- 172.21.15.100:0/2025222032 learned_addr learned my addr 172.21.15.100:0/2025222032 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a8affd700 1 -- 172.21.15.100:0/2025222032 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a7c0094f0 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a8affd700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c111540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c000c00 tx=0x7f0a7c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a7c0107c0 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a8c111c80 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f0a7c010db0 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a7c01ad00 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.513+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a8c1127f0 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.514+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f0a7c0193c0 con 0x7f0a8c100b20 2024-03-24T11:10:12.387 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.514+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a8c05f5f0 con 0x7f0a8c100b20 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.514+0000 7f0a737fe700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 0x7f0a74038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.514+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0a7c04a720 con 0x7f0a8c100b20 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.514+0000 7f0a8a7fc700 1 -- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 msgr2=0x7f0a74038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.514+0000 7f0a8a7fc700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 0x7f0a74038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.519+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0a7c0189a0 con 0x7f0a8c100b20 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.715+0000 7f0a8a7fc700 1 -- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 msgr2=0x7f0a74038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.715+0000 7f0a8a7fc700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 0x7f0a74038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.763+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0a8c112b30 con 0x7f0a8c100b20 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.764+0000 7f0a737fe700 1 -- 172.21.15.100:0/2025222032 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f0a7c018350 con 0x7f0a8c100b20 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 msgr2=0x7f0a74038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 0x7f0a74038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 msgr2=0x7f0a8c111540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c111540 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f0a7c000c00 tx=0x7f0a7c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:12.388 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 shutdown_connections 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7f0a740367e0 0x7f0a74038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 --2- 172.21.15.100:0/2025222032 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0a8c100b20 0x7f0a8c111540 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 >> 172.21.15.100:0/2025222032 conn(0x7f0a8c0fc500 msgr2=0x7f0a8c0fd0a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 shutdown_connections 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:11.766+0000 7f0a91791700 1 -- 172.21.15.100:0/2025222032 wait complete. 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for the mgr to restart... 2024-03-24T11:10:12.389 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr epoch 5... 2024-03-24T11:10:12.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:12 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2025222032' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: Active manager daemon smithi100.smxgyi restarted 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: Activating manager daemon smithi100.smxgyi 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: osdmap e2: 0 total, 0 up, 0 in 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: mgrmap e6: smithi100.smxgyi(active, starting, since 0.00327038s) 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi100.smxgyi", "id": "smithi100.smxgyi"}]: dispatch 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:10:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: Manager daemon smithi100.smxgyi is now available 2024-03-24T11:10:17.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:17.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:17.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:17.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:17.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:16 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:18.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:17 smithi100 ceph-mon[28941]: Found migration_current of "None". Setting to last migration. 2024-03-24T11:10:18.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:17 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:10:18.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:17 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/trash_purge_schedule"}]: dispatch 2024-03-24T11:10:18.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:17 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:18.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:17 smithi100 ceph-mon[28941]: mgrmap e7: smithi100.smxgyi(active, since 1.00505s) 2024-03-24T11:10:18.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:17 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 7, 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2fca6e700 1 Processor -- start 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2fca6e700 1 -- start start 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2fca6e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8105320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2fca6e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8105860 con 0x7fb2f8071320 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2f659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8105320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:18.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2f659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8105320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48282/0 (socket says 172.21.15.100:48282) 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2f659c700 1 -- 172.21.15.100:0/2925699869 learned_addr learned my addr 172.21.15.100:0/2925699869 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.817+0000 7fb2f659c700 1 -- 172.21.15.100:0/2925699869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f81059a0 con 0x7fb2f8071320 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.818+0000 7fb2f659c700 1 --2- 172.21.15.100:0/2925699869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8105320 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80098e0 tx=0x7fb2e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=980443701fc035d9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.818+0000 7fb2f559a700 1 -- 172.21.15.100:0/2925699869 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e800b860 con 0x7fb2f8071320 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.818+0000 7fb2f559a700 1 -- 172.21.15.100:0/2925699869 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb2e800be50 con 0x7fb2f8071320 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.818+0000 7fb2f559a700 1 -- 172.21.15.100:0/2925699869 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e8010380 con 0x7fb2f8071320 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 -- 172.21.15.100:0/2925699869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 msgr2=0x7fb2f8105320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/2925699869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8105320 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80098e0 tx=0x7fb2e8009250 comp rx=0 tx=0).stop 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 -- 172.21.15.100:0/2925699869 shutdown_connections 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/2925699869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8105320 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 -- 172.21.15.100:0/2925699869 >> 172.21.15.100:0/2925699869 conn(0x7fb2f80fcd50 msgr2=0x7fb2f80ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 -- 172.21.15.100:0/2925699869 shutdown_connections 2024-03-24T11:10:18.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 -- 172.21.15.100:0/2925699869 wait complete. 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 Processor -- start 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.819+0000 7fb2fca6e700 1 -- start start 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2fca6e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8111620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2fca6e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f8111b60 con 0x7fb2f8071320 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2f659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8111620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2f659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8111620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:48290/0 (socket says 172.21.15.100:48290) 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2f659c700 1 -- 172.21.15.100:0/4248552785 learned_addr learned my addr 172.21.15.100:0/4248552785 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2f659c700 1 -- 172.21.15.100:0/4248552785 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e80094f0 con 0x7fb2f8071320 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2f659c700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8111620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80090b0 tx=0x7fb2e8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e8010760 con 0x7fb2f8071320 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb2e8010d50 con 0x7fb2f8071320 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f8111d60 con 0x7fb2f8071320 2024-03-24T11:10:18.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f81128d0 con 0x7fb2f8071320 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.820+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e801ace0 con 0x7fb2f8071320 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.821+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 5) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb2e802d3f0 con 0x7fb2f8071320 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.821+0000 7fb2e6ffd700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.821+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 --> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2e00393a0 con 0x7fb2e00367e0 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.821+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fb2e804a090 con 0x7fb2f8071320 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.821+0000 7fb2f5d9b700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 msgr2=0x7fb2e0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:12.822+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:13.023+0000 7fb2f5d9b700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 msgr2=0x7fb2e0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:13.023+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:13.424+0000 7fb2f5d9b700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 msgr2=0x7fb2e0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:13.424+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:14.225+0000 7fb2f5d9b700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 msgr2=0x7fb2e0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:14.225+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T11:10:18.537 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:15.826+0000 7fb2f5d9b700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 msgr2=0x7fb2e0038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1648043756 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:15.826+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:16.919+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mgrmap(e 6) v1 ==== 43135+0+0 (secure 0 0 0) 0x7fb2e802d6f0 con 0x7fb2f8071320 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:16.919+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 msgr2=0x7fb2e0038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:16.919+0000 7fb2e6ffd700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.920+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mgrmap(e 7) v1 ==== 43262+0+0 (secure 0 0 0) 0x7fb2e8016d10 con 0x7fb2f8071320 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.920+0000 7fb2e6ffd700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2e0039fc0 0x7fb2e003c3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.920+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb2e00393a0 con 0x7fb2e0039fc0 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.920+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2e0039fc0 0x7fb2e003c3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.921+0000 7fb2f5d9b700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2e0039fc0 0x7fb2e003c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0039d0 tx=0x7fb2ec009350 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.921+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fb2e00393a0 con 0x7fb2e0039fc0 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.927+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb2f81121f0 con 0x7fb2e0039fc0 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.927+0000 7fb2e6ffd700 1 -- 172.21.15.100:0/4248552785 <== mgr.14120 v2:172.21.15.100:6800/1790161450 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7fb2f81121f0 con 0x7fb2e0039fc0 2024-03-24T11:10:18.538 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2e0039fc0 msgr2=0x7fb2e003c3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2e0039fc0 0x7fb2e003c3a0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7fb2ec0039d0 tx=0x7fb2ec009350 comp rx=0 tx=0).stop 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 msgr2=0x7fb2f8111620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8111620 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 crypto rx=0x7fb2e80090b0 tx=0x7fb2e8009250 comp rx=0 tx=0).stop 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 shutdown_connections 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f8071320 0x7fb2f8111620 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2e0039fc0 0x7fb2e003c3a0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 --2- 172.21.15.100:0/4248552785 >> [v2:172.21.15.100:6800/1648043756,v1:172.21.15.100:6801/1648043756] conn(0x7fb2e00367e0 0x7fb2e0038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 >> 172.21.15.100:0/4248552785 conn(0x7fb2f80fcd50 msgr2=0x7fb2f80fd990 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 shutdown_connections 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:17.928+0000 7fb2fca6e700 1 -- 172.21.15.100:0/4248552785 wait complete. 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:mgr epoch 5 is available 2024-03-24T11:10:18.539 INFO:teuthology.orchestra.run.smithi100.stdout:Setting orchestrator backend to cephadm... 2024-03-24T11:10:19.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:18 smithi100 ceph-mon[28941]: [24/Mar/2024:11:10:17] ENGINE Bus STARTING 2024-03-24T11:10:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:18 smithi100 ceph-mon[28941]: [24/Mar/2024:11:10:17] ENGINE Serving on https://172.21.15.100:7150 2024-03-24T11:10:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:18 smithi100 ceph-mon[28941]: [24/Mar/2024:11:10:17] ENGINE Bus STARTED 2024-03-24T11:10:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:18 smithi100 ceph-mon[28941]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:10:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:18 smithi100 ceph-mon[28941]: from='client.14124 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcac5dc2700 1 Processor -- start 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcac5dc2700 1 -- start start 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcac5dc2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0103100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcac5dc2700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac0103640 con 0x7fcac0102cf0 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcabf7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0103100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcabf7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0103100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55488/0 (socket says 172.21.15.100:55488) 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcabf7fe700 1 -- 172.21.15.100:0/905942423 learned_addr learned my addr 172.21.15.100:0/905942423 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:19.929 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcabf7fe700 1 -- 172.21.15.100:0/905942423 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcac0103780 con 0x7fcac0102cf0 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.986+0000 7fcabf7fe700 1 --2- 172.21.15.100:0/905942423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0103100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcab00098e0 tx=0x7fcab0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cde5268fc6fbf7aa server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcabe7fc700 1 -- 172.21.15.100:0/905942423 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcab000b8f0 con 0x7fcac0102cf0 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcabe7fc700 1 -- 172.21.15.100:0/905942423 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcab0011420 con 0x7fcac0102cf0 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcabe7fc700 1 -- 172.21.15.100:0/905942423 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcab0010540 con 0x7fcac0102cf0 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 -- 172.21.15.100:0/905942423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 msgr2=0x7fcac0103100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 --2- 172.21.15.100:0/905942423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0103100 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7fcab00098e0 tx=0x7fcab0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 -- 172.21.15.100:0/905942423 shutdown_connections 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 --2- 172.21.15.100:0/905942423 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0103100 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 -- 172.21.15.100:0/905942423 >> 172.21.15.100:0/905942423 conn(0x7fcac00fc500 msgr2=0x7fcac00fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 -- 172.21.15.100:0/905942423 shutdown_connections 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.987+0000 7fcac5dc2700 1 -- 172.21.15.100:0/905942423 wait complete. 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcac5dc2700 1 Processor -- start 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcac5dc2700 1 -- start start 2024-03-24T11:10:19.930 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcac5dc2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0113810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcac5dc2700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcac0113d50 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcabf7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0113810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcabf7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0113810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55504/0 (socket says 172.21.15.100:55504) 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcabf7fe700 1 -- 172.21.15.100:0/1349746008 learned_addr learned my addr 172.21.15.100:0/1349746008 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcabf7fe700 1 -- 172.21.15.100:0/1349746008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcab00094f0 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.988+0000 7fcabf7fe700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0113810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcab0011ef0 tx=0x7fcab0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.989+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcab0010910 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.989+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcac0113f50 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.989+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcab0010f00 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.989+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcab001ae50 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.989+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcac0114ac0 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.990+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fcab002d3f0 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.990+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcac005f5f0 con 0x7fcac0102cf0 2024-03-24T11:10:19.931 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.990+0000 7fcaa7fff700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcaa8036740 0x7fcaa8038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.990+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcab004a390 con 0x7fcac0102cf0 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.990+0000 7fcabeffd700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcaa8036740 0x7fcaa8038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.990+0000 7fcabeffd700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcaa8036740 0x7fcaa8038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009990 tx=0x7fcab4006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:18.995+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcab002b8e0 con 0x7fcac0102cf0 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.193+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fcac00fd8c0 con 0x7fcaa8036740 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.201+0000 7fcaa7fff700 1 -- 172.21.15.100:0/1349746008 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fcac00fd8c0 con 0x7fcaa8036740 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcaa8036740 msgr2=0x7fcaa8038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcaa8036740 0x7fcaa8038bf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcab4009990 tx=0x7fcab4006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 msgr2=0x7fcac0113810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0113810 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7fcab0011ef0 tx=0x7fcab0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 shutdown_connections 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcaa8036740 0x7fcaa8038bf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 --2- 172.21.15.100:0/1349746008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcac0102cf0 0x7fcac0113810 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:19.932 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 >> 172.21.15.100:0/1349746008 conn(0x7fcac00fc500 msgr2=0x7fcac00fd0f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:19.933 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 shutdown_connections 2024-03-24T11:10:19.933 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:19.203+0000 7fcac5dc2700 1 -- 172.21.15.100:0/1349746008 wait complete. 2024-03-24T11:10:20.240 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:19 smithi100 ceph-mon[28941]: mgrmap e8: smithi100.smxgyi(active, since 2s) 2024-03-24T11:10:20.240 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:19 smithi100 ceph-mon[28941]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:20.240 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:19 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:20.240 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:19 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:21.118 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:20 smithi100 ceph-mon[28941]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:21.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout value unchanged 2024-03-24T11:10:21.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f916d530700 1 Processor -- start 2024-03-24T11:10:21.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f916d530700 1 -- start start 2024-03-24T11:10:21.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f916d530700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168076110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:21.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f916d530700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9168078010 con 0x7f9168077c00 2024-03-24T11:10:21.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168076110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168076110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55508/0 (socket says 172.21.15.100:55508) 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.339+0000 7f9166ffd700 1 -- 172.21.15.100:0/3045389763 learned_addr learned my addr 172.21.15.100:0/3045389763 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.340+0000 7f9166ffd700 1 -- 172.21.15.100:0/3045389763 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9168076650 con 0x7f9168077c00 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.340+0000 7f9166ffd700 1 --2- 172.21.15.100:0/3045389763 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168076110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9158009e50 tx=0x7f9158009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b87fee9b4c1755e4 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.340+0000 7f9165ffb700 1 -- 172.21.15.100:0/3045389763 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91580092e0 con 0x7f9168077c00 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.340+0000 7f9165ffb700 1 -- 172.21.15.100:0/3045389763 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9158012430 con 0x7f9168077c00 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.340+0000 7f9165ffb700 1 -- 172.21.15.100:0/3045389763 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91580075c0 con 0x7f9168077c00 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- 172.21.15.100:0/3045389763 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 msgr2=0x7f9168076110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 --2- 172.21.15.100:0/3045389763 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168076110 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7f9158009e50 tx=0x7f9158009840 comp rx=0 tx=0).stop 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- 172.21.15.100:0/3045389763 shutdown_connections 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 --2- 172.21.15.100:0/3045389763 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168076110 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- 172.21.15.100:0/3045389763 >> 172.21.15.100:0/3045389763 conn(0x7f91680fde70 msgr2=0x7f9168100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- 172.21.15.100:0/3045389763 shutdown_connections 2024-03-24T11:10:21.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- 172.21.15.100:0/3045389763 wait complete. 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 Processor -- start 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- start start 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168113800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.341+0000 7f916d530700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9168113d40 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168113800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f9166ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168113800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55520/0 (socket says 172.21.15.100:55520) 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f9166ffd700 1 -- 172.21.15.100:0/1223765542 learned_addr learned my addr 172.21.15.100:0/1223765542 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f9166ffd700 1 -- 172.21.15.100:0/1223765542 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9158009ae0 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f9166ffd700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168113800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9158009fd0 tx=0x7f9158009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f91580092e0 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f9158007cc0 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9168113f40 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f915801c850 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.342+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9168114ab0 con 0x7f9168077c00 2024-03-24T11:10:21.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.343+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f9158019070 con 0x7f9168077c00 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.343+0000 7f914f7fe700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f91500367e0 0x7f9150038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.343+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f916805f5f0 con 0x7f9168077c00 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.343+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f9158049f70 con 0x7f9168077c00 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.343+0000 7f91667fc700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f91500367e0 0x7f9150038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.344+0000 7f91667fc700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f91500367e0 0x7f9150038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f915c009990 tx=0x7f915c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.348+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9158023b50 con 0x7f9168077c00 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.532+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f91680ff290 con 0x7f91500367e0 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.533+0000 7f914f7fe700 1 -- 172.21.15.100:0/1223765542 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f91680ff290 con 0x7f91500367e0 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f91500367e0 msgr2=0x7f9150038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:21.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f91500367e0 0x7f9150038c90 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f915c009990 tx=0x7f915c006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 msgr2=0x7f9168113800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168113800 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7f9158009fd0 tx=0x7f9158009840 comp rx=0 tx=0).stop 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 shutdown_connections 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f91500367e0 0x7f9150038c90 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 --2- 172.21.15.100:0/1223765542 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9168077c00 0x7f9168113800 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 >> 172.21.15.100:0/1223765542 conn(0x7f91680fde70 msgr2=0x7f91680feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 shutdown_connections 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:20.535+0000 7f916d530700 1 -- 172.21.15.100:0/1223765542 wait complete. 2024-03-24T11:10:21.130 INFO:teuthology.orchestra.run.smithi100.stdout:Generating ssh key... 2024-03-24T11:10:22.532 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f7107983700 1 Processor -- start 2024-03-24T11:10:22.532 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f7107983700 1 -- start start 2024-03-24T11:10:22.532 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f7107983700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f71001040c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:22.532 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f7107983700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7100104600 con 0x7f7100103cb0 2024-03-24T11:10:22.532 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f710571f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f71001040c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:22.532 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f710571f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f71001040c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55524/0 (socket says 172.21.15.100:55524) 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.541+0000 7f710571f700 1 -- 172.21.15.100:0/1262532204 learned_addr learned my addr 172.21.15.100:0/1262532204 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.542+0000 7f710571f700 1 -- 172.21.15.100:0/1262532204 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7100104740 con 0x7f7100103cb0 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.542+0000 7f710571f700 1 --2- 172.21.15.100:0/1262532204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f71001040c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0098e0 tx=0x7f70fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7e53352b18446ae6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.542+0000 7f70f7fff700 1 -- 172.21.15.100:0/1262532204 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70fc00b860 con 0x7f7100103cb0 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.542+0000 7f70f7fff700 1 -- 172.21.15.100:0/1262532204 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f70fc00be50 con 0x7f7100103cb0 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.542+0000 7f70f7fff700 1 -- 172.21.15.100:0/1262532204 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70fc010380 con 0x7f7100103cb0 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- 172.21.15.100:0/1262532204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 msgr2=0x7f71001040c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 --2- 172.21.15.100:0/1262532204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f71001040c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f70fc0098e0 tx=0x7f70fc009250 comp rx=0 tx=0).stop 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- 172.21.15.100:0/1262532204 shutdown_connections 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 --2- 172.21.15.100:0/1262532204 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f71001040c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- 172.21.15.100:0/1262532204 >> 172.21.15.100:0/1262532204 conn(0x7f71000fc4a0 msgr2=0x7f71000fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- 172.21.15.100:0/1262532204 shutdown_connections 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- 172.21.15.100:0/1262532204 wait complete. 2024-03-24T11:10:22.533 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 Processor -- start 2024-03-24T11:10:22.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- start start 2024-03-24T11:10:22.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f7100103450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:22.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.543+0000 7f7107983700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7100103990 con 0x7f7100103cb0 2024-03-24T11:10:22.534 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f710571f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f7100103450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f710571f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f7100103450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55534/0 (socket says 172.21.15.100:55534) 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f710571f700 1 -- 172.21.15.100:0/584918370 learned_addr learned my addr 172.21.15.100:0/584918370 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f710571f700 1 -- 172.21.15.100:0/584918370 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc0094f0 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f710571f700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f7100103450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00b440 tx=0x7f70fc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70fc010760 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f70fc010d50 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7100101a00 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70fc01ae20 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.544+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7100101ef0 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.545+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f70fc02e440 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.545+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710004bed0 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.545+0000 7f70f5ffb700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f70ec0367e0 0x7f70ec038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.545+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f70fc04a5b0 con 0x7f7100103cb0 2024-03-24T11:10:22.535 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.546+0000 7f7104f1e700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f70ec0367e0 0x7f70ec038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.546+0000 7f7104f1e700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f70ec0367e0 0x7f70ec038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009990 tx=0x7f70f0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.550+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f70fc018940 con 0x7f7100103cb0 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.735+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f7100102600 con 0x7f70ec0367e0 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.855+0000 7f70f5ffb700 1 -- 172.21.15.100:0/584918370 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7100102600 con 0x7f70ec0367e0 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f70ec0367e0 msgr2=0x7f70ec038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f70ec0367e0 0x7f70ec038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f70f0009990 tx=0x7f70f0006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 msgr2=0x7f7100103450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f7100103450 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f70fc00b440 tx=0x7f70fc009250 comp rx=0 tx=0).stop 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 shutdown_connections 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f70ec0367e0 0x7f70ec038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 --2- 172.21.15.100:0/584918370 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7100103cb0 0x7f7100103450 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.857+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 >> 172.21.15.100:0/584918370 conn(0x7f71000fc4a0 msgr2=0x7f71000fd020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.858+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 shutdown_connections 2024-03-24T11:10:22.536 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:21.858+0000 7f7107983700 1 -- 172.21.15.100:0/584918370 wait complete. 2024-03-24T11:10:23.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:22 smithi100 ceph-mon[28941]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:23.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:22 smithi100 ceph-mon[28941]: Generating ssh key... 2024-03-24T11:10:23.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:22 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:23.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:22 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:23.689 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1DXidWUrKTiQNp7+fb65m4J88vicxNAPrNHILyvoYa7gk7IPsv5bkNYAwKSP1kA98/uv/YoxPJW69CC2gZkRoLyt/QlznuUtPTiAy5zg+b9QsTcGWYNgSk3ByAGctz/+dHOrIgYU8EYg7x29g7nfxtuDHdfw9WO60dKyIbFOBc85MvImuO4NpyB3rjunPV3JOBhcDw3M7avYw6g5GNWLijZtKLluM5sjA2/yan0vGwObOxbtdzJMNP77bDafEoCq0+HQeZ5XJY4ueuCGrRfGCKmwQ5q1K7fvrx8iRmc/6fDF4LTCcOEmDMzCljmYbJXHcG1gvPXWumm9g8gX+S0019iHtjPDRedO9lDMi+3Vr3i/VYjLWOXbo8zfqfLy0/55hk/+I6+n3PXUt2LoTu/FFKq7O4tjMz9PdzODpCVgSpu+cqDeouXpa2J0mUXwrX/ozx1012i/kqtF1TOObEojTtHbNOMZQekwkjPBtLZKS7ysigtNDJ9B6nR/SQBxmP6U= ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f42670a6700 1 Processor -- start 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f42670a6700 1 -- start start 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f42670a6700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f42601076a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f42670a6700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260103090 con 0x7f4260076c20 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f4264e42700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f42601076a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f4264e42700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f42601076a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55546/0 (socket says 172.21.15.100:55546) 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.949+0000 7f4264e42700 1 -- 172.21.15.100:0/235867144 learned_addr learned my addr 172.21.15.100:0/235867144 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f4264e42700 1 -- 172.21.15.100:0/235867144 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4260107be0 con 0x7f4260076c20 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f4264e42700 1 --2- 172.21.15.100:0/235867144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f42601076a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f42540098e0 tx=0x7f4254009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=20ff6504ba8525f2 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f425f7fe700 1 -- 172.21.15.100:0/235867144 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425400b860 con 0x7f4260076c20 2024-03-24T11:10:23.690 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f425f7fe700 1 -- 172.21.15.100:0/235867144 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f425400be50 con 0x7f4260076c20 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f425f7fe700 1 -- 172.21.15.100:0/235867144 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4254010380 con 0x7f4260076c20 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f42670a6700 1 -- 172.21.15.100:0/235867144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 msgr2=0x7f42601076a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f42670a6700 1 --2- 172.21.15.100:0/235867144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f42601076a0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7f42540098e0 tx=0x7f4254009250 comp rx=0 tx=0).stop 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f42670a6700 1 -- 172.21.15.100:0/235867144 shutdown_connections 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f42670a6700 1 --2- 172.21.15.100:0/235867144 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f42601076a0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.950+0000 7f42670a6700 1 -- 172.21.15.100:0/235867144 >> 172.21.15.100:0/235867144 conn(0x7f42600fcd50 msgr2=0x7f42600ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f42670a6700 1 -- 172.21.15.100:0/235867144 shutdown_connections 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f42670a6700 1 -- 172.21.15.100:0/235867144 wait complete. 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f42670a6700 1 Processor -- start 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f42670a6700 1 -- start start 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f42670a6700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f4260113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f42670a6700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4260113c90 con 0x7f4260076c20 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f4264e42700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f4260113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:23.691 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f4264e42700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f4260113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55560/0 (socket says 172.21.15.100:55560) 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.951+0000 7f4264e42700 1 -- 172.21.15.100:0/1092156248 learned_addr learned my addr 172.21.15.100:0/1092156248 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.952+0000 7f4264e42700 1 -- 172.21.15.100:0/1092156248 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42540094f0 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.952+0000 7f4264e42700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f4260113750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4254003050 tx=0x7f4254009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.952+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4254010760 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.952+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f4254010d50 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.952+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4260113e90 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.952+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f425401ae20 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.953+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4260114a00 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.953+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f42540108c0 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.953+0000 7f425d7fa700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f42500367e0 0x7f4250038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.953+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f425404a4d0 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.953+0000 7f425ffff700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f42500367e0 0x7f4250038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.953+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f426005f5f0 con 0x7f4260076c20 2024-03-24T11:10:23.692 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.954+0000 7f425ffff700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f42500367e0 0x7f4250038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f424c009990 tx=0x7f424c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:22.958+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f425401a430 con 0x7f4260076c20 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.144+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f42600fe020 con 0x7f42500367e0 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.144+0000 7f425d7fa700 1 -- 172.21.15.100:0/1092156248 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f42600fe020 con 0x7f42500367e0 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f42500367e0 msgr2=0x7f4250038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f42500367e0 0x7f4250038c90 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7f424c009990 tx=0x7f424c006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 msgr2=0x7f4260113750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f4260113750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f4254003050 tx=0x7f4254009250 comp rx=0 tx=0).stop 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 shutdown_connections 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f42500367e0 0x7f4250038c90 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 --2- 172.21.15.100:0/1092156248 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4260076c20 0x7f4260113750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 >> 172.21.15.100:0/1092156248 conn(0x7f42600fcd50 msgr2=0x7f42600fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 shutdown_connections 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:23.147+0000 7f42670a6700 1 -- 172.21.15.100:0/1092156248 wait complete. 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:10:23.693 INFO:teuthology.orchestra.run.smithi100.stdout:Adding key to root@localhost authorized_keys... 2024-03-24T11:10:23.694 INFO:teuthology.orchestra.run.smithi100.stdout:Adding host smithi100... 2024-03-24T11:10:24.075 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:23 smithi100 ceph-mon[28941]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:24.980 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:24 smithi100 ceph-mon[28941]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi100", "addr": "172.21.15.100", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:26.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:25 smithi100 ceph-mon[28941]: Deploying cephadm binary to smithi100 2024-03-24T11:10:27.005 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Added host 'smithi100' with addr '172.21.15.100' 2024-03-24T11:10:27.005 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.099+0000 7fb2f5e40700 1 Processor -- start 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.099+0000 7fb2f5e40700 1 -- start start 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.099+0000 7fb2f5e40700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f01076f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.099+0000 7fb2f5e40700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f01030e0 con 0x7fb2f0076be0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f01076f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f01076f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55568/0 (socket says 172.21.15.100:55568) 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2ef7fe700 1 -- 172.21.15.100:0/3721762112 learned_addr learned my addr 172.21.15.100:0/3721762112 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2ef7fe700 1 -- 172.21.15.100:0/3721762112 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2f0107c30 con 0x7fb2f0076be0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2ef7fe700 1 --2- 172.21.15.100:0/3721762112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f01076f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00098e0 tx=0x7fb2e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a0e8baf01af18b9e server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2eeffd700 1 -- 172.21.15.100:0/3721762112 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e000b860 con 0x7fb2f0076be0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2eeffd700 1 -- 172.21.15.100:0/3721762112 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb2e000be50 con 0x7fb2f0076be0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.100+0000 7fb2eeffd700 1 -- 172.21.15.100:0/3721762112 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e0010380 con 0x7fb2f0076be0 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 -- 172.21.15.100:0/3721762112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 msgr2=0x7fb2f01076f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:27.006 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 --2- 172.21.15.100:0/3721762112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f01076f0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fb2e00098e0 tx=0x7fb2e0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 -- 172.21.15.100:0/3721762112 shutdown_connections 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 --2- 172.21.15.100:0/3721762112 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f01076f0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 -- 172.21.15.100:0/3721762112 >> 172.21.15.100:0/3721762112 conn(0x7fb2f00fcd80 msgr2=0x7fb2f00ff190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 -- 172.21.15.100:0/3721762112 shutdown_connections 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 -- 172.21.15.100:0/3721762112 wait complete. 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.101+0000 7fb2f5e40700 1 Processor -- start 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2f5e40700 1 -- start start 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2f5e40700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f0113780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2f5e40700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0113cc0 con 0x7fb2f0076be0 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f0113780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ef7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f0113780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55574/0 (socket says 172.21.15.100:55574) 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ef7fe700 1 -- 172.21.15.100:0/1082785689 learned_addr learned my addr 172.21.15.100:0/1082785689 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ef7fe700 1 -- 172.21.15.100:0/1082785689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e00094f0 con 0x7fb2f0076be0 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ef7fe700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f0113780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0006b00 tx=0x7fb2e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:27.007 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e00107c0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fb2e0010db0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e001acf0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.102+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f0113ec0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.103+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f0114a30 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.103+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fb2e00193c0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.104+0000 7fb2ecff9700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2dc0367e0 0x7fb2dc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.104+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb2e004a6c0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.104+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f005f5f0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.104+0000 7fb2e6ffd700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2dc0367e0 0x7fb2dc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.104+0000 7fb2e6ffd700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2dc0367e0 0x7fb2dc038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009990 tx=0x7fb2d8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.108+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb2e00196c0 con 0x7fb2f0076be0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:24.296+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi100", "addr": "172.21.15.100", "target": ["mon-mgr", ""]}) v1 -- 0x7fb2f00fe150 con 0x7fb2dc0367e0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.377+0000 7fb2ecff9700 1 -- 172.21.15.100:0/1082785689 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fb2f00fe150 con 0x7fb2dc0367e0 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.379+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2dc0367e0 msgr2=0x7fb2dc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:27.008 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.379+0000 7fb2f5e40700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2dc0367e0 0x7fb2dc038c90 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7fb2d8009990 tx=0x7fb2d8006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 msgr2=0x7fb2f0113780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f0113780 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7fb2e0006b00 tx=0x7fb2e0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 shutdown_connections 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fb2dc0367e0 0x7fb2dc038c90 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 --2- 172.21.15.100:0/1082785689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb2f0076be0 0x7fb2f0113780 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 >> 172.21.15.100:0/1082785689 conn(0x7fb2f00fcd80 msgr2=0x7fb2f00fd980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 shutdown_connections 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:26.380+0000 7fb2f5e40700 1 -- 172.21.15.100:0/1082785689 wait complete. 2024-03-24T11:10:27.009 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying mon service with default placement... 2024-03-24T11:10:27.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:27 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:27.637 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:27 smithi100 ceph-mon[28941]: Added host smithi100 2024-03-24T11:10:27.637 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:27 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:28.363 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-03-24T11:10:28.363 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.401+0000 7fc639e43700 1 Processor -- start 2024-03-24T11:10:28.363 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.401+0000 7fc639e43700 1 -- start start 2024-03-24T11:10:28.363 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.401+0000 7fc639e43700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc6341019e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.401+0000 7fc639e43700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc634101f20 con 0x7fc6341015d0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.401+0000 7fc6337fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc6341019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6337fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc6341019e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55586/0 (socket says 172.21.15.100:55586) 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6337fe700 1 -- 172.21.15.100:0/307387697 learned_addr learned my addr 172.21.15.100:0/307387697 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6337fe700 1 -- 172.21.15.100:0/307387697 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc634102060 con 0x7fc6341015d0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6337fe700 1 --2- 172.21.15.100:0/307387697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc6341019e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc62800bb80 tx=0x7fc62800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae5c69eeb0ee6a7b server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6327fc700 1 -- 172.21.15.100:0/307387697 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc62800dd00 con 0x7fc6341015d0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6327fc700 1 -- 172.21.15.100:0/307387697 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc62800de60 con 0x7fc6341015d0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.402+0000 7fc6327fc700 1 -- 172.21.15.100:0/307387697 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc6280188a0 con 0x7fc6341015d0 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 -- 172.21.15.100:0/307387697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 msgr2=0x7fc6341019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 --2- 172.21.15.100:0/307387697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc6341019e0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fc62800bb80 tx=0x7fc62800b4f0 comp rx=0 tx=0).stop 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 -- 172.21.15.100:0/307387697 shutdown_connections 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 --2- 172.21.15.100:0/307387697 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc6341019e0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:28.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 -- 172.21.15.100:0/307387697 >> 172.21.15.100:0/307387697 conn(0x7fc6340fcd50 msgr2=0x7fc6340ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 -- 172.21.15.100:0/307387697 shutdown_connections 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.403+0000 7fc639e43700 1 -- 172.21.15.100:0/307387697 wait complete. 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc639e43700 1 Processor -- start 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc639e43700 1 -- start start 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc639e43700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc63410f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc639e43700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc63410f8c0 con 0x7fc6341015d0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc6337fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc63410f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc6337fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc63410f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:55600/0 (socket says 172.21.15.100:55600) 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.404+0000 7fc6337fe700 1 -- 172.21.15.100:0/685613374 learned_addr learned my addr 172.21.15.100:0/685613374 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc6337fe700 1 -- 172.21.15.100:0/685613374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc62800b790 con 0x7fc6341015d0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc6337fe700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc63410f380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc628018600 tx=0x7fc62800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc628010bd0 con 0x7fc6341015d0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fc62801d050 con 0x7fc6341015d0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc63410fac0 con 0x7fc6341015d0 2024-03-24T11:10:28.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc628022670 con 0x7fc6341015d0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.405+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc634110630 con 0x7fc6341015d0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.406+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fc628022870 con 0x7fc6341015d0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.406+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc63405f5f0 con 0x7fc6341015d0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.406+0000 7fc61bfff700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc61c036790 0x7fc61c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.406+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc628014070 con 0x7fc6341015d0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.406+0000 7fc632ffd700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc61c036790 0x7fc61c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.407+0000 7fc632ffd700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc61c036790 0x7fc61c038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc624009990 tx=0x7fc624006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.411+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc62800e360 con 0x7fc6341015d0 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.636+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fc6341109e0 con 0x7fc61c036790 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.643+0000 7fc61bfff700 1 -- 172.21.15.100:0/685613374 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc6341109e0 con 0x7fc61c036790 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc61c036790 msgr2=0x7fc61c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc61c036790 0x7fc61c038c40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fc624009990 tx=0x7fc624006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:28.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 msgr2=0x7fc63410f380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc63410f380 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fc628018600 tx=0x7fc62800b4f0 comp rx=0 tx=0).stop 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 shutdown_connections 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc61c036790 0x7fc61c038c40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 --2- 172.21.15.100:0/685613374 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc6341015d0 0x7fc63410f380 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 >> 172.21.15.100:0/685613374 conn(0x7fc6340fcd50 msgr2=0x7fc634105b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 shutdown_connections 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:27.645+0000 7fc639e43700 1 -- 172.21.15.100:0/685613374 wait complete. 2024-03-24T11:10:28.367 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying mgr service with default placement... 2024-03-24T11:10:28.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:28 smithi100 ceph-mon[28941]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:28.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:28 smithi100 ceph-mon[28941]: Saving service mon spec with placement count:5 2024-03-24T11:10:28.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:28 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.788+0000 7fcff089c700 1 Processor -- start 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.788+0000 7fcff089c700 1 -- start start 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcff089c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec102b40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcff089c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec103080 con 0x7fcfec102730 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfea59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec102b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfea59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec102b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39718/0 (socket says 172.21.15.100:39718) 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfea59c700 1 -- 172.21.15.100:0/887879955 learned_addr learned my addr 172.21.15.100:0/887879955 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfea59c700 1 -- 172.21.15.100:0/887879955 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfec1031c0 con 0x7fcfec102730 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfea59c700 1 --2- 172.21.15.100:0/887879955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec102b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00098e0 tx=0x7fcfe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d428265a5f3ecfc5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:29.631 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfe959a700 1 -- 172.21.15.100:0/887879955 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe000b860 con 0x7fcfec102730 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfe959a700 1 -- 172.21.15.100:0/887879955 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcfe000be50 con 0x7fcfec102730 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.789+0000 7fcfe959a700 1 -- 172.21.15.100:0/887879955 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe0010380 con 0x7fcfec102730 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 -- 172.21.15.100:0/887879955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 msgr2=0x7fcfec102b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 --2- 172.21.15.100:0/887879955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec102b40 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7fcfe00098e0 tx=0x7fcfe0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 -- 172.21.15.100:0/887879955 shutdown_connections 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 --2- 172.21.15.100:0/887879955 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec102b40 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 -- 172.21.15.100:0/887879955 >> 172.21.15.100:0/887879955 conn(0x7fcfec0fdf10 msgr2=0x7fcfec100320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 -- 172.21.15.100:0/887879955 shutdown_connections 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 -- 172.21.15.100:0/887879955 wait complete. 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.790+0000 7fcff089c700 1 Processor -- start 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcff089c700 1 -- start start 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcff089c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec1115f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcff089c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfec111b30 con 0x7fcfec102730 2024-03-24T11:10:29.632 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfea59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec1115f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfea59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec1115f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39726/0 (socket says 172.21.15.100:39726) 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfea59c700 1 -- 172.21.15.100:0/1482322056 learned_addr learned my addr 172.21.15.100:0/1482322056 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfea59c700 1 -- 172.21.15.100:0/1482322056 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfe00094f0 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfea59c700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec1115f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0006fd0 tx=0x7fcfe0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe00107c0 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7fcfe0010db0 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfec111d30 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfe001ada0 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.791+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfec1128a0 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.792+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fcfe0019450 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.792+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfec1121c0 con 0x7fcfec102730 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.792+0000 7fcfdaffd700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcfd4036790 0x7fcfd4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:29.633 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.792+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fcfe0049d40 con 0x7fcfec102730 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.792+0000 7fcfe9d9b700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcfd4036790 0x7fcfd4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.793+0000 7fcfe9d9b700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcfd4036790 0x7fcfd4038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009990 tx=0x7fcfdc006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.797+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcfe0023cf0 con 0x7fcfec102730 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:28.998+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fcfec0ff2d0 con 0x7fcfd4036790 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.002+0000 7fcfdaffd700 1 -- 172.21.15.100:0/1482322056 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fcfec0ff2d0 con 0x7fcfd4036790 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcfd4036790 msgr2=0x7fcfd4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcfd4036790 0x7fcfd4038c40 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fcfdc009990 tx=0x7fcfdc006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 msgr2=0x7fcfec1115f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec1115f0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7fcfe0006fd0 tx=0x7fcfe0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 shutdown_connections 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fcfd4036790 0x7fcfd4038c40 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 --2- 172.21.15.100:0/1482322056 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcfec102730 0x7fcfec1115f0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 >> 172.21.15.100:0/1482322056 conn(0x7fcfec0fdf10 msgr2=0x7fcfec0feb00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 shutdown_connections 2024-03-24T11:10:29.634 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:29.004+0000 7fcff089c700 1 -- 172.21.15.100:0/1482322056 wait complete. 2024-03-24T11:10:29.635 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying crash service with default placement... 2024-03-24T11:10:30.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:30 smithi100 ceph-mon[28941]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:30.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:30 smithi100 ceph-mon[28941]: Saving service mgr spec with placement count:2 2024-03-24T11:10:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:30 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.364+0000 7f6aff2b9700 1 Processor -- start 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.364+0000 7f6aff2b9700 1 -- start start 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6aff2b9700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af8100fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6aff2b9700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af81014f0 con 0x7f6af8100ba0 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6afd055700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af8100fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6afd055700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af8100fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39728/0 (socket says 172.21.15.100:39728) 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6afd055700 1 -- 172.21.15.100:0/3387383075 learned_addr learned my addr 172.21.15.100:0/3387383075 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6afd055700 1 -- 172.21.15.100:0/3387383075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af8101630 con 0x7f6af8100ba0 2024-03-24T11:10:31.341 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6afd055700 1 --2- 172.21.15.100:0/3387383075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af8100fb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6af40098e0 tx=0x7f6af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7871ba0f83177ac5 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6aeffff700 1 -- 172.21.15.100:0/3387383075 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af400b860 con 0x7f6af8100ba0 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6aeffff700 1 -- 172.21.15.100:0/3387383075 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6af400be50 con 0x7f6af8100ba0 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.365+0000 7f6aeffff700 1 -- 172.21.15.100:0/3387383075 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af4010380 con 0x7f6af8100ba0 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 -- 172.21.15.100:0/3387383075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 msgr2=0x7f6af8100fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 --2- 172.21.15.100:0/3387383075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af8100fb0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f6af40098e0 tx=0x7f6af4009250 comp rx=0 tx=0).stop 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 -- 172.21.15.100:0/3387383075 shutdown_connections 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 --2- 172.21.15.100:0/3387383075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af8100fb0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 -- 172.21.15.100:0/3387383075 >> 172.21.15.100:0/3387383075 conn(0x7f6af80fc560 msgr2=0x7f6af80fe970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 -- 172.21.15.100:0/3387383075 shutdown_connections 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.366+0000 7f6aff2b9700 1 -- 172.21.15.100:0/3387383075 wait complete. 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6aff2b9700 1 Processor -- start 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6aff2b9700 1 -- start start 2024-03-24T11:10:31.342 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6aff2b9700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af81115c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6aff2b9700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6af8111b00 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6afd055700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af81115c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6afd055700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af81115c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39738/0 (socket says 172.21.15.100:39738) 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6afd055700 1 -- 172.21.15.100:0/584473672 learned_addr learned my addr 172.21.15.100:0/584473672 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6afd055700 1 -- 172.21.15.100:0/584473672 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6af40094f0 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.367+0000 7f6afd055700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af81115c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6af4006b00 tx=0x7f6af4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.368+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af4010760 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.368+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1044+0+0 (secure 0 0 0) 0x7f6af4010d50 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.368+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6af8111d00 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.368+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6af401ad40 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.368+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6af8112870 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.369+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f6af402e440 con 0x7f6af8100ba0 2024-03-24T11:10:31.343 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.369+0000 7f6aedffb700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f6ae40367e0 0x7f6ae4038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.369+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6af805f5f0 con 0x7f6af8100ba0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.369+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6af404b520 con 0x7f6af8100ba0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.369+0000 7f6afc854700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f6ae40367e0 0x7f6ae4038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.369+0000 7f6afc854700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f6ae40367e0 0x7f6ae4038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009990 tx=0x7f6ae8006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.374+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6af4018940 con 0x7f6af8100ba0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.570+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f6af80fd8e0 con 0x7f6ae40367e0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.574+0000 7f6aedffb700 1 -- 172.21.15.100:0/584473672 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6af80fd8e0 con 0x7f6ae40367e0 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f6ae40367e0 msgr2=0x7f6ae4038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f6ae40367e0 0x7f6ae4038c90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f6ae8009990 tx=0x7f6ae8006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 msgr2=0x7f6af81115c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af81115c0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f6af4006b00 tx=0x7f6af4009250 comp rx=0 tx=0).stop 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 shutdown_connections 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f6ae40367e0 0x7f6ae4038c90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:31.344 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 --2- 172.21.15.100:0/584473672 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6af8100ba0 0x7f6af81115c0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:31.345 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 >> 172.21.15.100:0/584473672 conn(0x7f6af80fc560 msgr2=0x7f6af80fd110 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:31.345 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 shutdown_connections 2024-03-24T11:10:31.345 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:30.576+0000 7f6aff2b9700 1 -- 172.21.15.100:0/584473672 wait complete. 2024-03-24T11:10:31.345 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying prometheus service with default placement... 2024-03-24T11:10:31.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:31 smithi100 ceph-mon[28941]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:31.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:31 smithi100 ceph-mon[28941]: Saving service crash spec with placement * 2024-03-24T11:10:31.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:31 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:31.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:31 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.735+0000 7f7333455700 1 Processor -- start 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.735+0000 7f7333455700 1 -- start start 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f7333455700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c1030d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f7333455700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c103610 con 0x7f732c071360 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f73311f1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c1030d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f73311f1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c1030d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39746/0 (socket says 172.21.15.100:39746) 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f73311f1700 1 -- 172.21.15.100:0/4247312652 learned_addr learned my addr 172.21.15.100:0/4247312652 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f73311f1700 1 -- 172.21.15.100:0/4247312652 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f732c103750 con 0x7f732c071360 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f73311f1700 1 --2- 172.21.15.100:0/4247312652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c1030d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f731c0098e0 tx=0x7f731c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10f51079ce458450 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.736+0000 7f7323fff700 1 -- 172.21.15.100:0/4247312652 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f731c00b860 con 0x7f732c071360 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7323fff700 1 -- 172.21.15.100:0/4247312652 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f731c00be90 con 0x7f732c071360 2024-03-24T11:10:32.757 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7323fff700 1 -- 172.21.15.100:0/4247312652 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f731c0103c0 con 0x7f732c071360 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 -- 172.21.15.100:0/4247312652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 msgr2=0x7f732c1030d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 --2- 172.21.15.100:0/4247312652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c1030d0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7f731c0098e0 tx=0x7f731c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 -- 172.21.15.100:0/4247312652 shutdown_connections 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 --2- 172.21.15.100:0/4247312652 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c1030d0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 -- 172.21.15.100:0/4247312652 >> 172.21.15.100:0/4247312652 conn(0x7f732c0fcd50 msgr2=0x7f732c0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 -- 172.21.15.100:0/4247312652 shutdown_connections 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.737+0000 7f7333455700 1 -- 172.21.15.100:0/4247312652 wait complete. 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f7333455700 1 Processor -- start 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f7333455700 1 -- start start 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f7333455700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c117c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f7333455700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f732c118170 con 0x7f732c071360 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f73311f1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c117c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:32.758 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f73311f1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c117c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39762/0 (socket says 172.21.15.100:39762) 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.738+0000 7f73311f1700 1 -- 172.21.15.100:0/197001299 learned_addr learned my addr 172.21.15.100:0/197001299 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f73311f1700 1 -- 172.21.15.100:0/197001299 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f731c0094f0 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f73311f1700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c117c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f731c0105b0 tx=0x7f731c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f731c010840 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f732c118370 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f731c010e70 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f731c01ae30 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.739+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f732c118ee0 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.740+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f732c05f5f0 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.740+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f731c0109a0 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.740+0000 7f7321ffb700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f7318036790 0x7f7318038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.740+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f731c04a1b0 con 0x7f732c071360 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.741+0000 7f73309f0700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f7318036790 0x7f7318038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:32.759 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.741+0000 7f73309f0700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f7318036790 0x7f7318038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7328009990 tx=0x7f7328006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.745+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f731c02b4d0 con 0x7f732c071360 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.981+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f732c119ac0 con 0x7f7318036790 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.986+0000 7f7321ffb700 1 -- 172.21.15.100:0/197001299 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f732c119ac0 con 0x7f7318036790 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f7318036790 msgr2=0x7f7318038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f7318036790 0x7f7318038c40 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f7328009990 tx=0x7f7328006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 msgr2=0x7f732c117c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c117c30 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7f731c0105b0 tx=0x7f731c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 shutdown_connections 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f7318036790 0x7f7318038c40 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 --2- 172.21.15.100:0/197001299 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f732c071360 0x7f732c117c30 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 >> 172.21.15.100:0/197001299 conn(0x7f732c0fcd50 msgr2=0x7f732c106c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 shutdown_connections 2024-03-24T11:10:32.760 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:31.989+0000 7f7333455700 1 -- 172.21.15.100:0/197001299 wait complete. 2024-03-24T11:10:32.761 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying grafana service with default placement... 2024-03-24T11:10:32.867 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:32 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:32.867 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:32 smithi100 ceph-mon[28941]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:32.867 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:32 smithi100 ceph-mon[28941]: Saving service prometheus spec with placement count:1 2024-03-24T11:10:32.867 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:32 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f57434fb700 1 Processor -- start 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f57434fb700 1 -- start start 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f57434fb700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1030e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f57434fb700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c103620 con 0x7f573c102cd0 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f5741297700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1030e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f5741297700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1030e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39774/0 (socket says 172.21.15.100:39774) 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.140+0000 7f5741297700 1 -- 172.21.15.100:0/3399328638 learned_addr learned my addr 172.21.15.100:0/3399328638 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f5741297700 1 -- 172.21.15.100:0/3399328638 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f573c103760 con 0x7f573c102cd0 2024-03-24T11:10:34.048 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f5741297700 1 --2- 172.21.15.100:0/3399328638 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1030e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098e0 tx=0x7f572c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=12ac7f322af3af42 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f5733fff700 1 -- 172.21.15.100:0/3399328638 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c00b8f0 con 0x7f573c102cd0 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f5733fff700 1 -- 172.21.15.100:0/3399328638 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f572c011420 con 0x7f573c102cd0 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f5733fff700 1 -- 172.21.15.100:0/3399328638 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c010540 con 0x7f573c102cd0 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f57434fb700 1 -- 172.21.15.100:0/3399328638 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 msgr2=0x7f573c1030e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f57434fb700 1 --2- 172.21.15.100:0/3399328638 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1030e0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f572c0098e0 tx=0x7f572c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f57434fb700 1 -- 172.21.15.100:0/3399328638 shutdown_connections 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f57434fb700 1 --2- 172.21.15.100:0/3399328638 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1030e0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.141+0000 7f57434fb700 1 -- 172.21.15.100:0/3399328638 >> 172.21.15.100:0/3399328638 conn(0x7f573c0fc4a0 msgr2=0x7f573c0fe8b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f57434fb700 1 -- 172.21.15.100:0/3399328638 shutdown_connections 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f57434fb700 1 -- 172.21.15.100:0/3399328638 wait complete. 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f57434fb700 1 Processor -- start 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f57434fb700 1 -- start start 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f57434fb700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f57434fb700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f573c113cf0 con 0x7f573c102cd0 2024-03-24T11:10:34.049 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f5741297700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f5741297700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39788/0 (socket says 172.21.15.100:39788) 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.142+0000 7f5741297700 1 -- 172.21.15.100:0/395821957 learned_addr learned my addr 172.21.15.100:0/395821957 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f5741297700 1 -- 172.21.15.100:0/395821957 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f572c0094f0 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f5741297700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1137b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f572c0090b0 tx=0x7f572c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c0107b0 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f572c010de0 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f572c01ae50 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f573c113ef0 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.143+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f573c114a60 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.144+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f572c010910 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.144+0000 7f5731ffb700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f5728036810 0x7f5728038cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.144+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f572c04a100 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.144+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f573c05f5f0 con 0x7f573c102cd0 2024-03-24T11:10:34.050 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.144+0000 7f5740a96700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f5728036810 0x7f5728038cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.144+0000 7f5740a96700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f5728036810 0x7f5728038cc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5738009990 tx=0x7f5738006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.149+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f572c02d3f0 con 0x7f573c102cd0 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.344+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f573c0fd7a0 con 0x7f5728036810 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.348+0000 7f5731ffb700 1 -- 172.21.15.100:0/395821957 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f573c0fd7a0 con 0x7f5728036810 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f5728036810 msgr2=0x7f5728038cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f5728036810 0x7f5728038cc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5738009990 tx=0x7f5738006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 msgr2=0x7f573c1137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1137b0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f572c0090b0 tx=0x7f572c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 shutdown_connections 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f5728036810 0x7f5728038cc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 --2- 172.21.15.100:0/395821957 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f573c102cd0 0x7f573c1137b0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.350+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 >> 172.21.15.100:0/395821957 conn(0x7f573c0fc4a0 msgr2=0x7f573c0fd070 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.351+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 shutdown_connections 2024-03-24T11:10:34.051 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:33.351+0000 7f57434fb700 1 -- 172.21.15.100:0/395821957 wait complete. 2024-03-24T11:10:34.052 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying node-exporter service with default placement... 2024-03-24T11:10:34.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:34 smithi100 ceph-mon[28941]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:34.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:34 smithi100 ceph-mon[28941]: Saving service grafana spec with placement count:1 2024-03-24T11:10:34.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:34 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:34.636 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:34 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5029501700 1 Processor -- start 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5029501700 1 -- start start 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5029501700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f50241019e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5029501700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5024101f20 con 0x7f50241015d0 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5022ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f50241019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5022ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f50241019e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39802/0 (socket says 172.21.15.100:39802) 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5022ffd700 1 -- 172.21.15.100:0/3060911715 learned_addr learned my addr 172.21.15.100:0/3060911715 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.492+0000 7f5022ffd700 1 -- 172.21.15.100:0/3060911715 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024102060 con 0x7f50241015d0 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5022ffd700 1 --2- 172.21.15.100:0/3060911715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f50241019e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50140098e0 tx=0x7f5014009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=5c196d9d2272ad19 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:35.364 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5021ffb700 1 -- 172.21.15.100:0/3060911715 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f501400b860 con 0x7f50241015d0 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5021ffb700 1 -- 172.21.15.100:0/3060911715 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f501400be90 con 0x7f50241015d0 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5021ffb700 1 -- 172.21.15.100:0/3060911715 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50140103c0 con 0x7f50241015d0 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5029501700 1 -- 172.21.15.100:0/3060911715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 msgr2=0x7f50241019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5029501700 1 --2- 172.21.15.100:0/3060911715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f50241019e0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7f50140098e0 tx=0x7f5014009250 comp rx=0 tx=0).stop 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5029501700 1 -- 172.21.15.100:0/3060911715 shutdown_connections 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5029501700 1 --2- 172.21.15.100:0/3060911715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f50241019e0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.493+0000 7f5029501700 1 -- 172.21.15.100:0/3060911715 >> 172.21.15.100:0/3060911715 conn(0x7f50240fcd50 msgr2=0x7f50240ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.494+0000 7f5029501700 1 -- 172.21.15.100:0/3060911715 shutdown_connections 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.494+0000 7f5029501700 1 -- 172.21.15.100:0/3060911715 wait complete. 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.494+0000 7f5029501700 1 Processor -- start 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.494+0000 7f5029501700 1 -- start start 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.494+0000 7f5029501700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f5024110500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.494+0000 7f5029501700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5024110a40 con 0x7f50241015d0 2024-03-24T11:10:35.365 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5022ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f5024110500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5022ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f5024110500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39814/0 (socket says 172.21.15.100:39814) 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5022ffd700 1 -- 172.21.15.100:0/668872824 learned_addr learned my addr 172.21.15.100:0/668872824 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5022ffd700 1 -- 172.21.15.100:0/668872824 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50140094f0 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5022ffd700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f5024110500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5014011c10 tx=0x7f5014009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50140118f0 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5014010b50 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5024110c40 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f501401adb0 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.495+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50241117b0 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.496+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f5014022030 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.496+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f502405f5f0 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.496+0000 7f500b7fe700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f500c036790 0x7f500c038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.496+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f501404c770 con 0x7f50241015d0 2024-03-24T11:10:35.366 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.496+0000 7f50227fc700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f500c036790 0x7f500c038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.497+0000 7f50227fc700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f500c036790 0x7f500c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5018009990 tx=0x7f5018006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.505+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5014023790 con 0x7f50241015d0 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.706+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f502405f5f0 con 0x7f500c036790 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.712+0000 7f500b7fe700 1 -- 172.21.15.100:0/668872824 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f502405f5f0 con 0x7f500c036790 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f500c036790 msgr2=0x7f500c038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f500c036790 0x7f500c038c40 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f5018009990 tx=0x7f5018006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 msgr2=0x7f5024110500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f5024110500 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7f5014011c10 tx=0x7f5014009250 comp rx=0 tx=0).stop 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 shutdown_connections 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f500c036790 0x7f500c038c40 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 --2- 172.21.15.100:0/668872824 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f50241015d0 0x7f5024110500 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 >> 172.21.15.100:0/668872824 conn(0x7f50240fcd50 msgr2=0x7f50240fd8d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 shutdown_connections 2024-03-24T11:10:35.367 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:34.714+0000 7f5029501700 1 -- 172.21.15.100:0/668872824 wait complete. 2024-03-24T11:10:35.368 INFO:teuthology.orchestra.run.smithi100.stdout:Deploying alertmanager service with default placement... 2024-03-24T11:10:35.734 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:35 smithi100 ceph-mon[28941]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:35.734 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:35 smithi100 ceph-mon[28941]: Saving service node-exporter spec with placement * 2024-03-24T11:10:35.734 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:35 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:35.734 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:35 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:36.722 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-03-24T11:10:36.722 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14c51c700 1 Processor -- start 2024-03-24T11:10:36.722 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14c51c700 1 -- start start 2024-03-24T11:10:36.722 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14c51c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe1441053b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:36.722 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14c51c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1441058f0 con 0x7fe144076bb0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14a2b8700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe1441053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14a2b8700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe1441053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39830/0 (socket says 172.21.15.100:39830) 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14a2b8700 1 -- 172.21.15.100:0/3939648765 learned_addr learned my addr 172.21.15.100:0/3939648765 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.830+0000 7fe14a2b8700 1 -- 172.21.15.100:0/3939648765 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe144105a30 con 0x7fe144076bb0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14a2b8700 1 --2- 172.21.15.100:0/3939648765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe1441053b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1400098e0 tx=0x7fe140009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=1159484633cb09ae server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe1492b6700 1 -- 172.21.15.100:0/3939648765 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe14000b860 con 0x7fe144076bb0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe1492b6700 1 -- 172.21.15.100:0/3939648765 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe14000be90 con 0x7fe144076bb0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe1492b6700 1 -- 172.21.15.100:0/3939648765 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1400103c0 con 0x7fe144076bb0 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 -- 172.21.15.100:0/3939648765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 msgr2=0x7fe1441053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 --2- 172.21.15.100:0/3939648765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe1441053b0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fe1400098e0 tx=0x7fe140009250 comp rx=0 tx=0).stop 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 -- 172.21.15.100:0/3939648765 shutdown_connections 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 --2- 172.21.15.100:0/3939648765 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe1441053b0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 -- 172.21.15.100:0/3939648765 >> 172.21.15.100:0/3939648765 conn(0x7fe1440fce40 msgr2=0x7fe1440ff270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:36.723 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 -- 172.21.15.100:0/3939648765 shutdown_connections 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.831+0000 7fe14c51c700 1 -- 172.21.15.100:0/3939648765 wait complete. 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14c51c700 1 Processor -- start 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14c51c700 1 -- start start 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14c51c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe144111610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14c51c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe144111b50 con 0x7fe144076bb0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14a2b8700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe144111610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14a2b8700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe144111610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39832/0 (socket says 172.21.15.100:39832) 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14a2b8700 1 -- 172.21.15.100:0/790804770 learned_addr learned my addr 172.21.15.100:0/790804770 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14a2b8700 1 -- 172.21.15.100:0/790804770 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1400094f0 con 0x7fe144076bb0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.832+0000 7fe14a2b8700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe144111610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1400105b0 tx=0x7fe140009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.833+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe140010840 con 0x7fe144076bb0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.833+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe144111d50 con 0x7fe144076bb0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.833+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe140010e70 con 0x7fe144076bb0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.833+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe14001ae30 con 0x7fe144076bb0 2024-03-24T11:10:36.724 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.833+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1441128c0 con 0x7fe144076bb0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.834+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe14405f5f0 con 0x7fe144076bb0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.834+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fe14001a430 con 0x7fe144076bb0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.834+0000 7fe13affd700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fe1300367e0 0x7fe130038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.834+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fe14004a590 con 0x7fe144076bb0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.834+0000 7fe149ab7700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fe1300367e0 0x7fe130038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.834+0000 7fe149ab7700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fe1300367e0 0x7fe130038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe134009990 tx=0x7fe134006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:35.838+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe14002c930 con 0x7fe144076bb0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.053+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1440fe270 con 0x7fe1300367e0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.060+0000 7fe13affd700 1 -- 172.21.15.100:0/790804770 <== mgr.14120 v2:172.21.15.100:6800/1790161450 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fe1440fe270 con 0x7fe1300367e0 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fe1300367e0 msgr2=0x7fe130038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fe1300367e0 0x7fe130038c90 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7fe134009990 tx=0x7fe134006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 msgr2=0x7fe144111610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe144111610 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fe1400105b0 tx=0x7fe140009250 comp rx=0 tx=0).stop 2024-03-24T11:10:36.725 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 shutdown_connections 2024-03-24T11:10:36.726 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fe1300367e0 0x7fe130038c90 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:36.726 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 --2- 172.21.15.100:0/790804770 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe144076bb0 0x7fe144111610 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:36.726 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 >> 172.21.15.100:0/790804770 conn(0x7fe1440fce40 msgr2=0x7fe1440fdaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:36.726 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 shutdown_connections 2024-03-24T11:10:36.726 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:36.062+0000 7fe14c51c700 1 -- 172.21.15.100:0/790804770 wait complete. 2024-03-24T11:10:37.301 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:37 smithi100 ceph-mon[28941]: from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:37.301 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:37 smithi100 ceph-mon[28941]: Saving service alertmanager spec with placement count:1 2024-03-24T11:10:37.301 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:37 smithi100 ceph-mon[28941]: from='mgr.14120 172.21.15.100:0/3741861778' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:38.146 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.178+0000 7fd5b9194700 1 Processor -- start 2024-03-24T11:10:38.146 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.178+0000 7fd5b9194700 1 -- start start 2024-03-24T11:10:38.146 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.178+0000 7fd5b9194700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4102ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:38.146 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.178+0000 7fd5b9194700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4103530 con 0x7fd5b4102be0 2024-03-24T11:10:38.146 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4102ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:38.146 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4102ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39840/0 (socket says 172.21.15.100:39840) 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b2d9d700 1 -- 172.21.15.100:0/766073273 learned_addr learned my addr 172.21.15.100:0/766073273 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b2d9d700 1 -- 172.21.15.100:0/766073273 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5b4103670 con 0x7fd5b4102be0 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b2d9d700 1 --2- 172.21.15.100:0/766073273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4102ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098e0 tx=0x7fd5a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2fc1ca6af899f837 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b259c700 1 -- 172.21.15.100:0/766073273 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a400b860 con 0x7fd5b4102be0 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b259c700 1 -- 172.21.15.100:0/766073273 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5a400be90 con 0x7fd5b4102be0 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.179+0000 7fd5b259c700 1 -- 172.21.15.100:0/766073273 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a40103c0 con 0x7fd5b4102be0 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 -- 172.21.15.100:0/766073273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 msgr2=0x7fd5b4102ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 --2- 172.21.15.100:0/766073273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4102ff0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40098e0 tx=0x7fd5a4009250 comp rx=0 tx=0).stop 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 -- 172.21.15.100:0/766073273 shutdown_connections 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 --2- 172.21.15.100:0/766073273 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4102ff0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 -- 172.21.15.100:0/766073273 >> 172.21.15.100:0/766073273 conn(0x7fd5b40fc500 msgr2=0x7fd5b40fe910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 -- 172.21.15.100:0/766073273 shutdown_connections 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 -- 172.21.15.100:0/766073273 wait complete. 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 Processor -- start 2024-03-24T11:10:38.147 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.180+0000 7fd5b9194700 1 -- start start 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b9194700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b9194700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5b4111ac0 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b2d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:39846/0 (socket says 172.21.15.100:39846) 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b2d9d700 1 -- 172.21.15.100:0/3632885504 learned_addr learned my addr 172.21.15.100:0/3632885504 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b2d9d700 1 -- 172.21.15.100:0/3632885504 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5a40094f0 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b2d9d700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4111580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40090b0 tx=0x7fd5a4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a4010840 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd5a4010e70 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5b4111cc0 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd5a4024e40 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.181+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5b4112830 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.182+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7fd5a40109a0 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.182+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5b405f5f0 con 0x7fd5b4102be0 2024-03-24T11:10:38.148 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.182+0000 7fd5abfff700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fd5a0036740 0x7fd5a0038bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.182+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd5a404a790 con 0x7fd5b4102be0 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.182+0000 7fd5aa59c700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fd5a0036740 0x7fd5a0038bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.183+0000 7fd5aa59c700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fd5a0036740 0x7fd5a0038bf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009990 tx=0x7fd59c006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.187+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd5a402c520 con 0x7fd5b4102be0 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.374+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fd5b404bed0 con 0x7fd5b4102be0 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.380+0000 7fd5abfff700 1 -- 172.21.15.100:0/3632885504 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd5a402c340 con 0x7fd5b4102be0 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fd5a0036740 msgr2=0x7fd5a0038bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fd5a0036740 0x7fd5a0038bf0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fd59c009990 tx=0x7fd59c006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 msgr2=0x7fd5b4111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4111580 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fd5a40090b0 tx=0x7fd5a4009250 comp rx=0 tx=0).stop 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 shutdown_connections 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fd5a0036740 0x7fd5a0038bf0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 --2- 172.21.15.100:0/3632885504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd5b4102be0 0x7fd5b4111580 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.382+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 >> 172.21.15.100:0/3632885504 conn(0x7fd5b40fc500 msgr2=0x7fd5b40fd3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:38.149 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.383+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 shutdown_connections 2024-03-24T11:10:38.150 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:37.383+0000 7fd5b9194700 1 -- 172.21.15.100:0/3632885504 wait complete. 2024-03-24T11:10:38.541 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:38 smithi100 ceph-mon[28941]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:10:38.541 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:38 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3632885504' entity='client.admin' 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.950+0000 7f33dd21a700 1 Processor -- start 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.950+0000 7f33dd21a700 1 -- start start 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.950+0000 7f33dd21a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8101a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.950+0000 7f33dd21a700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d8101f70 con 0x7f33d8101620 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8101a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8101a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40052/0 (socket says 172.21.15.100:40052) 2024-03-24T11:10:39.788 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d6d9d700 1 -- 172.21.15.100:0/272098578 learned_addr learned my addr 172.21.15.100:0/272098578 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d6d9d700 1 -- 172.21.15.100:0/272098578 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33d81020b0 con 0x7f33d8101620 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d6d9d700 1 --2- 172.21.15.100:0/272098578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8101a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33c800bb80 tx=0x7f33c800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2e4582d1f209f977 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d659c700 1 -- 172.21.15.100:0/272098578 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c800dd00 con 0x7f33d8101620 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d659c700 1 -- 172.21.15.100:0/272098578 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33c800de60 con 0x7f33d8101620 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.951+0000 7f33d659c700 1 -- 172.21.15.100:0/272098578 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c8018920 con 0x7f33d8101620 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- 172.21.15.100:0/272098578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 msgr2=0x7f33d8101a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 --2- 172.21.15.100:0/272098578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8101a30 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f33c800bb80 tx=0x7f33c800b4f0 comp rx=0 tx=0).stop 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- 172.21.15.100:0/272098578 shutdown_connections 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 --2- 172.21.15.100:0/272098578 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8101a30 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- 172.21.15.100:0/272098578 >> 172.21.15.100:0/272098578 conn(0x7f33d80fcdc0 msgr2=0x7f33d80ff1d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- 172.21.15.100:0/272098578 shutdown_connections 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- 172.21.15.100:0/272098578 wait complete. 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 Processor -- start 2024-03-24T11:10:39.789 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- start start 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8113730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.952+0000 7f33dd21a700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33d8113c70 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33d6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8113730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33d6d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8113730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40056/0 (socket says 172.21.15.100:40056) 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33d6d9d700 1 -- 172.21.15.100:0/4027887315 learned_addr learned my addr 172.21.15.100:0/4027887315 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33d6d9d700 1 -- 172.21.15.100:0/4027887315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33c800b790 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33d6d9d700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8113730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f33c80106a0 tx=0x7f33c800b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c800dc70 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f33c8010b80 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33d8113e70 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33c8021550 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.953+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33d81149e0 con 0x7f33d8101620 2024-03-24T11:10:39.790 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.954+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f33c8021810 con 0x7f33d8101620 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.954+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33d805f5f0 con 0x7f33d8101620 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.954+0000 7f33cf7fe700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f33c4036790 0x7f33c4038c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.954+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f33c8014070 con 0x7f33d8101620 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.954+0000 7f33cffff700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f33c4036790 0x7f33c4038c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.955+0000 7f33cffff700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f33c4036790 0x7f33c4038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009990 tx=0x7f33c0006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:38.959+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f33c800ed90 con 0x7f33d8101620 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.141+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f33d804bed0 con 0x7f33d8101620 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.144+0000 7f33cf7fe700 1 -- 172.21.15.100:0/4027887315 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7f33c800e360 con 0x7f33d8101620 2024-03-24T11:10:39.791 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f33c4036790 msgr2=0x7f33c4038c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f33c4036790 0x7f33c4038c40 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f33c0009990 tx=0x7f33c0006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 msgr2=0x7f33d8113730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8113730 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f33c80106a0 tx=0x7f33c800b4f0 comp rx=0 tx=0).stop 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 shutdown_connections 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f33c4036790 0x7f33c4038c40 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 --2- 172.21.15.100:0/4027887315 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f33d8101620 0x7f33d8113730 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.146+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 >> 172.21.15.100:0/4027887315 conn(0x7f33d80fcdc0 msgr2=0x7f33d8105b10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:39.792 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.147+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 shutdown_connections 2024-03-24T11:10:39.793 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:39.147+0000 7f33dd21a700 1 -- 172.21.15.100:0/4027887315 wait complete. 2024-03-24T11:10:39.793 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling the dashboard module... 2024-03-24T11:10:40.171 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:40 smithi100 ceph-mon[28941]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:10:40.171 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:40 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/4027887315' entity='client.admin' 2024-03-24T11:10:41.536 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:41 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1069178210' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-03-24T11:10:41.798 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.169+0000 7f78f7de1700 1 Processor -- start 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f7de1700 1 -- start start 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f7de1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f0102b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f7de1700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0103090 con 0x7f78f0102740 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f5b7d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f0102b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f5b7d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f0102b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40068/0 (socket says 172.21.15.100:40068) 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f5b7d700 1 -- 172.21.15.100:0/1965691849 learned_addr learned my addr 172.21.15.100:0/1965691849 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f5b7d700 1 -- 172.21.15.100:0/1965691849 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78f01031d0 con 0x7f78f0102740 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.170+0000 7f78f5b7d700 1 --2- 172.21.15.100:0/1965691849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f0102b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78e00098e0 tx=0x7f78e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9b03118c60e062a9 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f4b7b700 1 -- 172.21.15.100:0/1965691849 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78e000b860 con 0x7f78f0102740 2024-03-24T11:10:41.804 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f4b7b700 1 -- 172.21.15.100:0/1965691849 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78e000be90 con 0x7f78f0102740 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f4b7b700 1 -- 172.21.15.100:0/1965691849 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78e00103c0 con 0x7f78f0102740 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 -- 172.21.15.100:0/1965691849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 msgr2=0x7f78f0102b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 --2- 172.21.15.100:0/1965691849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f0102b50 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f78e00098e0 tx=0x7f78e0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 -- 172.21.15.100:0/1965691849 shutdown_connections 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 --2- 172.21.15.100:0/1965691849 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f0102b50 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 -- 172.21.15.100:0/1965691849 >> 172.21.15.100:0/1965691849 conn(0x7f78f00fdea0 msgr2=0x7f78f01002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 -- 172.21.15.100:0/1965691849 shutdown_connections 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.171+0000 7f78f7de1700 1 -- 172.21.15.100:0/1965691849 wait complete. 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f7de1700 1 Processor -- start 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f7de1700 1 -- start start 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f7de1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f01115e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f7de1700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78f0111b20 con 0x7f78f0102740 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f5b7d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f01115e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:41.805 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f5b7d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f01115e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40070/0 (socket says 172.21.15.100:40070) 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f5b7d700 1 -- 172.21.15.100:0/1069178210 learned_addr learned my addr 172.21.15.100:0/1069178210 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f5b7d700 1 -- 172.21.15.100:0/1069178210 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78e00094f0 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78f5b7d700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f01115e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78e0011c10 tx=0x7f78e0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78e0010890 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.172+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f78e0010ec0 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.173+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f78f0111d20 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.173+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f78e00243f0 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.173+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f78f0112890 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.173+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 8) v1 ==== 43368+0+0 (secure 0 0 0) 0x7f78e001a430 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.173+0000 7f78e67fc700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f78dc0367e0 0x7f78dc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.174+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f78f005f5f0 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.174+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f78e004a290 con 0x7f78f0102740 2024-03-24T11:10:41.806 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.174+0000 7f78f537c700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f78dc0367e0 0x7f78dc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.174+0000 7f78f537c700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f78dc0367e0 0x7f78dc038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009990 tx=0x7f78ec006e40 comp rx=0 tx=0).ready entity=mgr.14120 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.178+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f78e002b8c0 con 0x7f78f0102740 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:40.441+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f78f004bed0 con 0x7f78f0102740 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.147+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f78e002d3f0 con 0x7f78f0102740 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.148+0000 7f78e67fc700 1 -- 172.21.15.100:0/1069178210 <== mon.0 v2:172.21.15.100:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7f78e0018940 con 0x7f78f0102740 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f78dc0367e0 msgr2=0x7f78dc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f78dc0367e0 0x7f78dc038c90 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f78ec009990 tx=0x7f78ec006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 msgr2=0x7f78f01115e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f01115e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f78e0011c10 tx=0x7f78e0009250 comp rx=0 tx=0).stop 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 shutdown_connections 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f78dc0367e0 0x7f78dc038c90 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 --2- 172.21.15.100:0/1069178210 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f78f0102740 0x7f78f01115e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.151+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 >> 172.21.15.100:0/1069178210 conn(0x7f78f00fdea0 msgr2=0x7f78f00fea30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:41.807 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.152+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 shutdown_connections 2024-03-24T11:10:41.808 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:41.152+0000 7f78f7de1700 1 -- 172.21.15.100:0/1069178210 wait complete. 2024-03-24T11:10:42.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:42 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1069178210' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-03-24T11:10:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:42 smithi100 ceph-mon[28941]: mgrmap e9: smithi100.smxgyi(active, since 24s) 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "available": true, 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "active_name": "smithi100.smxgyi", 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc184732700 1 Processor -- start 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc184732700 1 -- start start 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc184732700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c102b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc184732700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c103070 con 0x7fc17c102720 2024-03-24T11:10:43.061 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc1824ce700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c102b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc1824ce700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c102b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40094/0 (socket says 172.21.15.100:40094) 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.201+0000 7fc1824ce700 1 -- 172.21.15.100:0/162031956 learned_addr learned my addr 172.21.15.100:0/162031956 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.202+0000 7fc1824ce700 1 -- 172.21.15.100:0/162031956 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc17c1031b0 con 0x7fc17c102720 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.202+0000 7fc1824ce700 1 --2- 172.21.15.100:0/162031956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c102b30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc1780098e0 tx=0x7fc178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b99a14e0ff3bf79c server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.202+0000 7fc1814cc700 1 -- 172.21.15.100:0/162031956 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc17800b860 con 0x7fc17c102720 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.202+0000 7fc1814cc700 1 -- 172.21.15.100:0/162031956 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc17800be90 con 0x7fc17c102720 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.202+0000 7fc1814cc700 1 -- 172.21.15.100:0/162031956 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1780103c0 con 0x7fc17c102720 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 -- 172.21.15.100:0/162031956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 msgr2=0x7fc17c102b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 --2- 172.21.15.100:0/162031956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c102b30 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fc1780098e0 tx=0x7fc178009250 comp rx=0 tx=0).stop 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 -- 172.21.15.100:0/162031956 shutdown_connections 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 --2- 172.21.15.100:0/162031956 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c102b30 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 -- 172.21.15.100:0/162031956 >> 172.21.15.100:0/162031956 conn(0x7fc17c0fdee0 msgr2=0x7fc17c1002f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:43.062 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 -- 172.21.15.100:0/162031956 shutdown_connections 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 -- 172.21.15.100:0/162031956 wait complete. 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 Processor -- start 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.203+0000 7fc184732700 1 -- start start 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc184732700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c111580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc184732700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc17c111ac0 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc1824ce700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c111580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc1824ce700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c111580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40102/0 (socket says 172.21.15.100:40102) 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc1824ce700 1 -- 172.21.15.100:0/2228750701 learned_addr learned my addr 172.21.15.100:0/2228750701 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc1824ce700 1 -- 172.21.15.100:0/2228750701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1780094f0 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc1824ce700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c111580 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc178011680 tx=0x7fc178009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc178010890 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc178010ec0 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1780243f0 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc17c111cc0 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.204+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc17c112830 con 0x7fc17c102720 2024-03-24T11:10:43.063 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.205+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fc17802d3f0 con 0x7fc17c102720 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.205+0000 7fc172ffd700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 0x7fc168038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.205+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc17804a240 con 0x7fc17c102720 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.205+0000 7fc181ccd700 1 -- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 msgr2=0x7fc168038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.205+0000 7fc181ccd700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 0x7fc168038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.205+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc17c05f5f0 con 0x7fc17c102720 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.210+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc17802b8e0 con 0x7fc17c102720 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.406+0000 7fc181ccd700 1 -- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 msgr2=0x7fc168038ce0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.406+0000 7fc181ccd700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 0x7fc168038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.471+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fc17c112a70 con 0x7fc17c102720 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.471+0000 7fc172ffd700 1 -- 172.21.15.100:0/2228750701 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fc17802c350 con 0x7fc17c102720 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 msgr2=0x7fc168038ce0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 0x7fc168038ce0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 msgr2=0x7fc17c111580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:43.064 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c111580 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fc178011680 tx=0x7fc178009250 comp rx=0 tx=0).stop 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 shutdown_connections 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7fc168036830 0x7fc168038ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 --2- 172.21.15.100:0/2228750701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc17c102720 0x7fc17c111580 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 >> 172.21.15.100:0/2228750701 conn(0x7fc17c0fdee0 msgr2=0x7fc17c0fea80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 shutdown_connections 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:42.473+0000 7fc184732700 1 -- 172.21.15.100:0/2228750701 wait complete. 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for the mgr to restart... 2024-03-24T11:10:43.065 INFO:teuthology.orchestra.run.smithi100.stdout:Waiting for mgr epoch 9... 2024-03-24T11:10:43.380 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:43 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2228750701' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: Active manager daemon smithi100.smxgyi restarted 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: Activating manager daemon smithi100.smxgyi 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: osdmap e3: 0 total, 0 up, 0 in 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: mgrmap e10: smithi100.smxgyi(active, starting, since 0.00295925s) 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi100.smxgyi", "id": "smithi100.smxgyi"}]: dispatch 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: Manager daemon smithi100.smxgyi is now available 2024-03-24T11:10:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:48.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:10:48.976 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:10:48.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/trash_purge_schedule"}]: dispatch 2024-03-24T11:10:48.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: [24/Mar/2024:11:10:47] ENGINE Bus STARTING 2024-03-24T11:10:48.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: [24/Mar/2024:11:10:47] ENGINE Serving on https://172.21.15.100:7150 2024-03-24T11:10:48.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: [24/Mar/2024:11:10:47] ENGINE Bus STARTED 2024-03-24T11:10:48.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:48.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:48 smithi100 ceph-mon[28941]: mgrmap e11: smithi100.smxgyi(active, since 1.00562s) 2024-03-24T11:10:49.270 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout { 2024-03-24T11:10:49.270 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout "initialized": true 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout } 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e0a915700 1 Processor -- start 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e0a915700 1 -- start start 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e0a915700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041019e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e0a915700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04101f20 con 0x7f1e041015d0 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041019e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041019e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40108/0 (socket says 172.21.15.100:40108) 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e03fff700 1 -- 172.21.15.100:0/977218989 learned_addr learned my addr 172.21.15.100:0/977218989 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.484+0000 7f1e03fff700 1 -- 172.21.15.100:0/977218989 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04102060 con 0x7f1e041015d0 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e03fff700 1 --2- 172.21.15.100:0/977218989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041019e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1df400bb80 tx=0x7f1df400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b2e6e871dfb44ba7 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e02ffd700 1 -- 172.21.15.100:0/977218989 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df400dd00 con 0x7f1e041015d0 2024-03-24T11:10:49.271 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e02ffd700 1 -- 172.21.15.100:0/977218989 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1df400de60 con 0x7f1e041015d0 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e02ffd700 1 -- 172.21.15.100:0/977218989 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df4018920 con 0x7f1e041015d0 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e0a915700 1 -- 172.21.15.100:0/977218989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 msgr2=0x7f1e041019e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e0a915700 1 --2- 172.21.15.100:0/977218989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041019e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f1df400bb80 tx=0x7f1df400b4f0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e0a915700 1 -- 172.21.15.100:0/977218989 shutdown_connections 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e0a915700 1 --2- 172.21.15.100:0/977218989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041019e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.485+0000 7f1e0a915700 1 -- 172.21.15.100:0/977218989 >> 172.21.15.100:0/977218989 conn(0x7f1e040fcd50 msgr2=0x7f1e040ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e0a915700 1 -- 172.21.15.100:0/977218989 shutdown_connections 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e0a915700 1 -- 172.21.15.100:0/977218989 wait complete. 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e0a915700 1 Processor -- start 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e0a915700 1 -- start start 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e0a915700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e0a915700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e04113cf0 con 0x7f1e041015d0 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e03fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:40124/0 (socket says 172.21.15.100:40124) 2024-03-24T11:10:49.272 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.486+0000 7f1e03fff700 1 -- 172.21.15.100:0/1132237116 learned_addr learned my addr 172.21.15.100:0/1132237116 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e03fff700 1 -- 172.21.15.100:0/1132237116 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1df400b790 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e03fff700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041137b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1df40106a0 tx=0x7f1df400b4f0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df4018e90 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e04113ef0 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1df400dc70 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1df4021550 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.487+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e04114a60 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.488+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 9) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f1df40217d0 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.488+0000 7f1e00ff9700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.488+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 --> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1df4018e90 con 0x7f1dec0367e0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.488+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1df4014070 con 0x7f1e041015d0 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.488+0000 7f1e037fe700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 msgr2=0x7f1dec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.488+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:10:49.273 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.689+0000 7f1e037fe700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 msgr2=0x7f1dec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:43.689+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:44.090+0000 7f1e037fe700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 msgr2=0x7f1dec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:44.090+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:44.891+0000 7f1e037fe700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 msgr2=0x7f1dec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:44.891+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:46.493+0000 7f1e037fe700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 msgr2=0x7f1dec038c90 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1790161450 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:46.493+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 3.200000 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:47.650+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mgrmap(e 10) v1 ==== 43148+0+0 (secure 0 0 0) 0x7f1df40229c0 con 0x7f1e041015d0 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:47.650+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 msgr2=0x7f1dec038c90 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:47.650+0000 7f1e00ff9700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.651+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7f1df401f940 con 0x7f1e041015d0 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.651+0000 7f1e00ff9700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f1dec039fe0 0x7f1dec03c3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.651+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1df4018e90 con 0x7f1dec039fe0 2024-03-24T11:10:49.274 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.652+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f1dec039fe0 0x7f1dec03c3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.652+0000 7f1e037fe700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f1dec039fe0 0x7f1dec03c3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1df80039d0 tx=0x7f1df80092f0 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.653+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f1df4018e90 con 0x7f1dec039fe0 2024-03-24T11:10:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f1e04114380 con 0x7f1dec039fe0 2024-03-24T11:10:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e00ff9700 1 -- 172.21.15.100:0/1132237116 <== mgr.14162 v2:172.21.15.100:6800/1493018813 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f1e04114380 con 0x7f1dec039fe0 2024-03-24T11:10:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f1dec039fe0 msgr2=0x7f1dec03c3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:49.275 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f1dec039fe0 0x7f1dec03c3c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f1df80039d0 tx=0x7f1df80092f0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 msgr2=0x7f1e041137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041137b0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f1df40106a0 tx=0x7f1df400b4f0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 shutdown_connections 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f1dec039fe0 0x7f1dec03c3c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1e041015d0 0x7f1e041137b0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 --2- 172.21.15.100:0/1132237116 >> [v2:172.21.15.100:6800/1790161450,v1:172.21.15.100:6801/1790161450] conn(0x7f1dec0367e0 0x7f1dec038c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 >> 172.21.15.100:0/1132237116 conn(0x7f1e040fcd50 msgr2=0x7f1e04105b70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 shutdown_connections 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:48.659+0000 7f1e0a915700 1 -- 172.21.15.100:0/1132237116 wait complete. 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:mgr epoch 9 is available 2024-03-24T11:10:49.276 INFO:teuthology.orchestra.run.smithi100.stdout:Generating a dashboard self-signed certificate... 2024-03-24T11:10:49.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:49 smithi100 ceph-mon[28941]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-03-24T11:10:49.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:49 smithi100 ceph-mon[28941]: from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-03-24T11:10:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:49 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.694+0000 7fa50297d700 1 Processor -- start 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.694+0000 7fa50297d700 1 -- start start 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa50297d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc101dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa50297d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc102390 con 0x7fa4fc1019b0 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa4fbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc101dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa4fbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc101dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53898/0 (socket says 172.21.15.100:53898) 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa4fbfff700 1 -- 172.21.15.100:0/1914266774 learned_addr learned my addr 172.21.15.100:0/1914266774 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa4fbfff700 1 -- 172.21.15.100:0/1914266774 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4fc1024d0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa4fbfff700 1 --2- 172.21.15.100:0/1914266774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc101dc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0098e0 tx=0x7fa4ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=83ef6ee4a8f74e6d server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.695+0000 7fa4faffd700 1 -- 172.21.15.100:0/1914266774 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ec00b860 con 0x7fa4fc1019b0 2024-03-24T11:10:50.775 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa4faffd700 1 -- 172.21.15.100:0/1914266774 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4ec00be90 con 0x7fa4fc1019b0 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa4faffd700 1 -- 172.21.15.100:0/1914266774 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ec0103c0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 -- 172.21.15.100:0/1914266774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 msgr2=0x7fa4fc101dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 --2- 172.21.15.100:0/1914266774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc101dc0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0098e0 tx=0x7fa4ec009250 comp rx=0 tx=0).stop 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 -- 172.21.15.100:0/1914266774 shutdown_connections 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 --2- 172.21.15.100:0/1914266774 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc101dc0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 -- 172.21.15.100:0/1914266774 >> 172.21.15.100:0/1914266774 conn(0x7fa4fc0fc040 msgr2=0x7fa4fc0fe470 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 -- 172.21.15.100:0/1914266774 shutdown_connections 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.696+0000 7fa50297d700 1 -- 172.21.15.100:0/1914266774 wait complete. 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa50297d700 1 Processor -- start 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa50297d700 1 -- start start 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa50297d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc1137e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa50297d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4fc113d20 con 0x7fa4fc1019b0 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa4fbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc1137e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa4fbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc1137e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53912/0 (socket says 172.21.15.100:53912) 2024-03-24T11:10:50.776 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa4fbfff700 1 -- 172.21.15.100:0/2055232793 learned_addr learned my addr 172.21.15.100:0/2055232793 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.697+0000 7fa4fbfff700 1 -- 172.21.15.100:0/2055232793 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4ec0094f0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.698+0000 7fa4fbfff700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc1137e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0094c0 tx=0x7fa4ec009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.698+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ec010890 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.698+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa4ec010ec0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.698+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4fc113f20 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.698+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4ec0243f0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.698+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4fc114a90 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.699+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 11) v1 ==== 43275+0+0 (secure 0 0 0) 0x7fa4ec02d3f0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.699+0000 7fa4f8ff9700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fa4e40366b0 0x7fa4e4038b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.699+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4fc05f5f0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.699+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fa4ec04b280 con 0x7fa4fc1019b0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.699+0000 7fa4fb7fe700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fa4e40366b0 0x7fa4e4038b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.699+0000 7fa4fb7fe700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fa4e40366b0 0x7fa4e4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0009990 tx=0x7fa4f0006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:50.777 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.704+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa4ec02b8e0 con 0x7fa4fc1019b0 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:49.967+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa4fc115610 con 0x7fa4e40366b0 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.062+0000 7fa4f8ff9700 1 -- 172.21.15.100:0/2055232793 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa4fc115610 con 0x7fa4e40366b0 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fa4e40366b0 msgr2=0x7fa4e4038b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fa4e40366b0 0x7fa4e4038b60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fa4f0009990 tx=0x7fa4f0006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 msgr2=0x7fa4fc1137e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc1137e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fa4ec0094c0 tx=0x7fa4ec009250 comp rx=0 tx=0).stop 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 shutdown_connections 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fa4e40366b0 0x7fa4e4038b60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 --2- 172.21.15.100:0/2055232793 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa4fc1019b0 0x7fa4fc1137e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.064+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 >> 172.21.15.100:0/2055232793 conn(0x7fa4fc0fc040 msgr2=0x7fa4fc10a650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.065+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 shutdown_connections 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:50.065+0000 7fa50297d700 1 -- 172.21.15.100:0/2055232793 wait complete. 2024-03-24T11:10:50.778 INFO:teuthology.orchestra.run.smithi100.stdout:Creating initial admin user... 2024-03-24T11:10:51.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:51 smithi100 ceph-mon[28941]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:51.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:51 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:51.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:51 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:51.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:51 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:51.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:51 smithi100 ceph-mon[28941]: mgrmap e12: smithi100.smxgyi(active, since 2s) 2024-03-24T11:10:52.970 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:52 smithi100 ceph-mon[28941]: from='client.14176 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:10:52.971 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:52 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:10:52.979 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$0NXmDvmH0YlUit7X3qljneuQHpLW7chCKkYFU7qGShoa2zGK0/RkC", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1711278651, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-03-24T11:10:52.979 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e3768700 1 Processor -- start 2024-03-24T11:10:52.980 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e3768700 1 -- start start 2024-03-24T11:10:52.980 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e3768700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc107700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:52.980 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e3768700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc1030f0 con 0x7f18dc076bc0 2024-03-24T11:10:52.980 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e1504700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc107700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:52.980 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e1504700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc107700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53928/0 (socket says 172.21.15.100:53928) 2024-03-24T11:10:52.980 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.203+0000 7f18e1504700 1 -- 172.21.15.100:0/3797962308 learned_addr learned my addr 172.21.15.100:0/3797962308 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18e1504700 1 -- 172.21.15.100:0/3797962308 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18dc107c40 con 0x7f18dc076bc0 2024-03-24T11:10:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18e1504700 1 --2- 172.21.15.100:0/3797962308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc107700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18d80098e0 tx=0x7f18d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=71f518535c4d7fc6 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18d3fff700 1 -- 172.21.15.100:0/3797962308 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18d800b860 con 0x7f18dc076bc0 2024-03-24T11:10:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18d3fff700 1 -- 172.21.15.100:0/3797962308 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18d800be90 con 0x7f18dc076bc0 2024-03-24T11:10:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18d3fff700 1 -- 172.21.15.100:0/3797962308 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18d80103c0 con 0x7f18dc076bc0 2024-03-24T11:10:52.981 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18e3768700 1 -- 172.21.15.100:0/3797962308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 msgr2=0x7f18dc107700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:52.982 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.204+0000 7f18e3768700 1 --2- 172.21.15.100:0/3797962308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc107700 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f18d80098e0 tx=0x7f18d8009250 comp rx=0 tx=0).stop 2024-03-24T11:10:52.982 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 -- 172.21.15.100:0/3797962308 shutdown_connections 2024-03-24T11:10:52.982 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 --2- 172.21.15.100:0/3797962308 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc107700 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:52.982 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 -- 172.21.15.100:0/3797962308 >> 172.21.15.100:0/3797962308 conn(0x7f18dc0fcd50 msgr2=0x7f18dc0ff160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:52.982 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 -- 172.21.15.100:0/3797962308 shutdown_connections 2024-03-24T11:10:52.982 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 -- 172.21.15.100:0/3797962308 wait complete. 2024-03-24T11:10:52.983 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 Processor -- start 2024-03-24T11:10:52.983 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 -- start start 2024-03-24T11:10:52.983 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc113730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:52.983 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e3768700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18dc113c70 con 0x7f18dc076bc0 2024-03-24T11:10:52.983 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e1504700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc113730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:52.983 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e1504700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc113730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53930/0 (socket says 172.21.15.100:53930) 2024-03-24T11:10:52.984 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.205+0000 7f18e1504700 1 -- 172.21.15.100:0/2793948790 learned_addr learned my addr 172.21.15.100:0/2793948790 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:52.984 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18e1504700 1 -- 172.21.15.100:0/2793948790 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18d80094f0 con 0x7f18dc076bc0 2024-03-24T11:10:52.984 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18e1504700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc113730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18d80094c0 tx=0x7f18d8009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:52.984 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18d8010890 con 0x7f18dc076bc0 2024-03-24T11:10:52.984 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f18d8010ec0 con 0x7f18dc076bc0 2024-03-24T11:10:52.984 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f18d801aea0 con 0x7f18dc076bc0 2024-03-24T11:10:52.985 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18dc113ed0 con 0x7f18dc076bc0 2024-03-24T11:10:52.985 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.206+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18dc1149e0 con 0x7f18dc076bc0 2024-03-24T11:10:52.985 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.207+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f18d801a430 con 0x7f18dc076bc0 2024-03-24T11:10:52.985 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.207+0000 7f18d1ffb700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f18c80367b0 0x7f18c8038c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:52.985 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.207+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18dc05f5f0 con 0x7f18dc076bc0 2024-03-24T11:10:52.985 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.207+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f18d804a340 con 0x7f18dc076bc0 2024-03-24T11:10:52.986 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.207+0000 7f18e0d03700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f18c80367b0 0x7f18c8038c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:52.986 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.208+0000 7f18e0d03700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f18c80367b0 0x7f18c8038c60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009990 tx=0x7f18cc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:52.986 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.212+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f18d802b6b0 con 0x7f18dc076bc0 2024-03-24T11:10:52.986 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.437+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f18dc05f410 con 0x7f18c80367b0 2024-03-24T11:10:52.986 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.672+0000 7f18d1ffb700 1 -- 172.21.15.100:0/2793948790 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f18dc05f410 con 0x7f18c80367b0 2024-03-24T11:10:52.986 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f18c80367b0 msgr2=0x7f18c8038c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:52.987 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f18c80367b0 0x7f18c8038c60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f18cc009990 tx=0x7f18cc006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:52.987 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 msgr2=0x7f18dc113730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:52.987 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc113730 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f18d80094c0 tx=0x7f18d8009250 comp rx=0 tx=0).stop 2024-03-24T11:10:52.987 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 shutdown_connections 2024-03-24T11:10:52.987 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f18c80367b0 0x7f18c8038c60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:52.987 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 --2- 172.21.15.100:0/2793948790 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f18dc076bc0 0x7f18dc113730 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:52.988 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 >> 172.21.15.100:0/2793948790 conn(0x7f18dc0fcd50 msgr2=0x7f18dc0fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:52.988 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 shutdown_connections 2024-03-24T11:10:52.988 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:51.674+0000 7f18e3768700 1 -- 172.21.15.100:0/2793948790 wait complete. 2024-03-24T11:10:52.988 INFO:teuthology.orchestra.run.smithi100.stdout:Fetching dashboard port number... 2024-03-24T11:10:53.851 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:53 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/4079822688' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stdout 8443 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be8e71700 1 Processor -- start 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be8e71700 1 -- start start 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be8e71700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4077220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be8e71700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be40790d0 con 0x7f0be4078cc0 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4077220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4077220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53932/0 (socket says 172.21.15.100:53932) 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.360+0000 7f0be259c700 1 -- 172.21.15.100:0/2980329156 learned_addr learned my addr 172.21.15.100:0/2980329156 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:54.126 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be259c700 1 -- 172.21.15.100:0/2980329156 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0be4077760 con 0x7f0be4078cc0 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be259c700 1 --2- 172.21.15.100:0/2980329156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4077220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40098e0 tx=0x7f0bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=bce786cb14e9e849 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be159a700 1 -- 172.21.15.100:0/2980329156 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd400b860 con 0x7f0be4078cc0 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be159a700 1 -- 172.21.15.100:0/2980329156 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bd400be90 con 0x7f0be4078cc0 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be159a700 1 -- 172.21.15.100:0/2980329156 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd40103c0 con 0x7f0be4078cc0 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be8e71700 1 -- 172.21.15.100:0/2980329156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 msgr2=0x7f0be4077220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.361+0000 7f0be8e71700 1 --2- 172.21.15.100:0/2980329156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4077220 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40098e0 tx=0x7f0bd4009250 comp rx=0 tx=0).stop 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 -- 172.21.15.100:0/2980329156 shutdown_connections 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 --2- 172.21.15.100:0/2980329156 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4077220 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 -- 172.21.15.100:0/2980329156 >> 172.21.15.100:0/2980329156 conn(0x7f0be40fdea0 msgr2=0x7f0be41002d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 -- 172.21.15.100:0/2980329156 shutdown_connections 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 -- 172.21.15.100:0/2980329156 wait complete. 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 Processor -- start 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 -- start start 2024-03-24T11:10:54.127 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4113850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be8e71700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0be4113d90 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4113850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4113850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53938/0 (socket says 172.21.15.100:53938) 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.362+0000 7f0be259c700 1 -- 172.21.15.100:0/4079822688 learned_addr learned my addr 172.21.15.100:0/4079822688 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0be259c700 1 -- 172.21.15.100:0/4079822688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd40094f0 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0be259c700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4113850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40105b0 tx=0x7f0bd4009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd4010840 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0bd4010e70 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0be4113f90 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0bd401ae30 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.363+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0be4114b00 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.364+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7f0bd40109a0 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.364+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0be405f5f0 con 0x7f0be4078cc0 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.364+0000 7f0bd2ffd700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0bcc0367e0 0x7f0bcc038c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:54.128 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.364+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0bd404a310 con 0x7f0be4078cc0 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.364+0000 7f0be1d9b700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0bcc0367e0 0x7f0bcc038c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.364+0000 7f0be1d9b700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0bcc0367e0 0x7f0bcc038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009990 tx=0x7f0bd8006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.369+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0bd402b4d0 con 0x7f0be4078cc0 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.556+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f0be404bed0 con 0x7f0be4078cc0 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.556+0000 7f0bd2ffd700 1 -- 172.21.15.100:0/4079822688 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f0bd402b2f0 con 0x7f0be4078cc0 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0bcc0367e0 msgr2=0x7f0bcc038c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0bcc0367e0 0x7f0bcc038c90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f0bd8009990 tx=0x7f0bd8006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 msgr2=0x7f0be4113850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4113850 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f0bd40105b0 tx=0x7f0bd4009250 comp rx=0 tx=0).stop 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 shutdown_connections 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0bcc0367e0 0x7f0bcc038c90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 --2- 172.21.15.100:0/4079822688 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0be4078cc0 0x7f0be4113850 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 >> 172.21.15.100:0/4079822688 conn(0x7f0be40fdea0 msgr2=0x7f0be40feaf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 shutdown_connections 2024-03-24T11:10:54.129 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:53.558+0000 7f0be8e71700 1 -- 172.21.15.100:0/4079822688 wait complete. 2024-03-24T11:10:54.133 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-03-24T11:10:54.133 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout disabled 2024-03-24T11:10:54.140 INFO:teuthology.orchestra.run.smithi100.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-03-24T11:10:54.140 INFO:teuthology.orchestra.run.smithi100.stdout:systemctl: stdout inactive 2024-03-24T11:10:54.140 INFO:teuthology.orchestra.run.smithi100.stdout:firewalld.service is not enabled 2024-03-24T11:10:54.140 INFO:teuthology.orchestra.run.smithi100.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-03-24T11:10:54.142 INFO:teuthology.orchestra.run.smithi100.stdout:Ceph Dashboard is now available at: 2024-03-24T11:10:54.142 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:54.142 INFO:teuthology.orchestra.run.smithi100.stdout: URL: https://smithi100.front.sepia.ceph.com:8443/ 2024-03-24T11:10:54.142 INFO:teuthology.orchestra.run.smithi100.stdout: User: admin 2024-03-24T11:10:54.143 INFO:teuthology.orchestra.run.smithi100.stdout: Password: hu948fpl21 2024-03-24T11:10:54.143 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:54.143 INFO:teuthology.orchestra.run.smithi100.stdout:Saving cluster configuration to /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config directory 2024-03-24T11:10:54.143 INFO:teuthology.orchestra.run.smithi100.stdout:Enabling autotune for osd_memory_target 2024-03-24T11:10:55.334 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.580+0000 7fd263aa4700 1 Processor -- start 2024-03-24T11:10:55.334 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.580+0000 7fd263aa4700 1 -- start start 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.580+0000 7fd263aa4700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c102ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.580+0000 7fd263aa4700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c103000 con 0x7fd25c1026b0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.580+0000 7fd261840700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c102ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd261840700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c102ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53942/0 (socket says 172.21.15.100:53942) 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd261840700 1 -- 172.21.15.100:0/2589366352 learned_addr learned my addr 172.21.15.100:0/2589366352 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd261840700 1 -- 172.21.15.100:0/2589366352 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd25c103140 con 0x7fd25c1026b0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd261840700 1 --2- 172.21.15.100:0/2589366352 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c102ac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd254009e50 tx=0x7fd254009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=d51fcd92e1671098 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd26083e700 1 -- 172.21.15.100:0/2589366352 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2540092e0 con 0x7fd25c1026b0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd26083e700 1 -- 172.21.15.100:0/2589366352 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd254012430 con 0x7fd25c1026b0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.581+0000 7fd26083e700 1 -- 172.21.15.100:0/2589366352 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2540075a0 con 0x7fd25c1026b0 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 -- 172.21.15.100:0/2589366352 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 msgr2=0x7fd25c102ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 --2- 172.21.15.100:0/2589366352 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c102ac0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd254009e50 tx=0x7fd254009840 comp rx=0 tx=0).stop 2024-03-24T11:10:55.335 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 -- 172.21.15.100:0/2589366352 shutdown_connections 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 --2- 172.21.15.100:0/2589366352 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c102ac0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 -- 172.21.15.100:0/2589366352 >> 172.21.15.100:0/2589366352 conn(0x7fd25c0fde70 msgr2=0x7fd25c100280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 -- 172.21.15.100:0/2589366352 shutdown_connections 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 -- 172.21.15.100:0/2589366352 wait complete. 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 Processor -- start 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 -- start start 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.582+0000 7fd263aa4700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c077ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd263aa4700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25c078000 con 0x7fd25c1026b0 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd261840700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c077ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd261840700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c077ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53950/0 (socket says 172.21.15.100:53950) 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd261840700 1 -- 172.21.15.100:0/869225841 learned_addr learned my addr 172.21.15.100:0/869225841 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:55.336 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd261840700 1 -- 172.21.15.100:0/869225841 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd254009ae0 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd261840700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c077ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd254009fd0 tx=0x7fd254009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd2540092e0 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd254007ce0 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25c076070 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd254025990 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.583+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25c076560 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.584+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 12) v1 ==== 43381+0+0 (secure 0 0 0) 0x7fd254019050 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.584+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd25c04bed0 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.584+0000 7fd2527fc700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fd248036830 0x7fd248038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.584+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd25404a330 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.584+0000 7fd26103f700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fd248036830 0x7fd248038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.585+0000 7fd26103f700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fd248036830 0x7fd248038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009990 tx=0x7fd24c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.589+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd25402d350 con 0x7fd25c1026b0 2024-03-24T11:10:55.337 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.663+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7fd254007a30 con 0x7fd25c1026b0 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.775+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fd25c0769e0 con 0x7fd25c1026b0 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.775+0000 7fd2527fc700 1 -- 172.21.15.100:0/869225841 <== mon.0 v2:172.21.15.100:3300/0 8 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fd254017070 con 0x7fd25c1026b0 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fd248036830 msgr2=0x7fd248038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fd248036830 0x7fd248038ce0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fd24c009990 tx=0x7fd24c006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 msgr2=0x7fd25c077ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c077ac0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7fd254009fd0 tx=0x7fd254009840 comp rx=0 tx=0).stop 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 shutdown_connections 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7fd248036830 0x7fd248038ce0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 --2- 172.21.15.100:0/869225841 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd25c1026b0 0x7fd25c077ac0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 >> 172.21.15.100:0/869225841 conn(0x7fd25c0fde70 msgr2=0x7fd25c0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 shutdown_connections 2024-03-24T11:10:55.338 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:54.777+0000 7fd263aa4700 1 -- 172.21.15.100:0/869225841 wait complete. 2024-03-24T11:10:55.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:55 smithi100 ceph-mon[28941]: mgrmap e13: smithi100.smxgyi(active, since 7s) 2024-03-24T11:10:56.583 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6628125700 1 Processor -- start 2024-03-24T11:10:56.583 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6628125700 1 -- start start 2024-03-24T11:10:56.583 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6628125700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f6620102b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6628125700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66201030c0 con 0x7f6620102770 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6625ec1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f6620102b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6625ec1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f6620102b80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53966/0 (socket says 172.21.15.100:53966) 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6625ec1700 1 -- 172.21.15.100:0/47611950 learned_addr learned my addr 172.21.15.100:0/47611950 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.735+0000 7f6625ec1700 1 -- 172.21.15.100:0/47611950 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6620103200 con 0x7f6620102770 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6625ec1700 1 --2- 172.21.15.100:0/47611950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f6620102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f661c0098e0 tx=0x7f661c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f63f5f311c0b65e3 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6624ebf700 1 -- 172.21.15.100:0/47611950 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661c00b860 con 0x7f6620102770 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6624ebf700 1 -- 172.21.15.100:0/47611950 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f661c00be90 con 0x7f6620102770 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6624ebf700 1 -- 172.21.15.100:0/47611950 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661c0103c0 con 0x7f6620102770 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 -- 172.21.15.100:0/47611950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 msgr2=0x7f6620102b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:56.584 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 --2- 172.21.15.100:0/47611950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f6620102b80 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f661c0098e0 tx=0x7f661c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 -- 172.21.15.100:0/47611950 shutdown_connections 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 --2- 172.21.15.100:0/47611950 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f6620102b80 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 -- 172.21.15.100:0/47611950 >> 172.21.15.100:0/47611950 conn(0x7f66200fde90 msgr2=0x7f6620100300 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 -- 172.21.15.100:0/47611950 shutdown_connections 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.736+0000 7f6628125700 1 -- 172.21.15.100:0/47611950 wait complete. 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6628125700 1 Processor -- start 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6628125700 1 -- start start 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6628125700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f66201115d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6628125700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6620111b10 con 0x7f6620102770 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6625ec1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f66201115d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6625ec1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f66201115d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53972/0 (socket says 172.21.15.100:53972) 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6625ec1700 1 -- 172.21.15.100:0/615150739 learned_addr learned my addr 172.21.15.100:0/615150739 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6625ec1700 1 -- 172.21.15.100:0/615150739 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661c0094f0 con 0x7f6620102770 2024-03-24T11:10:56.585 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.737+0000 7f6625ec1700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f66201115d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f661c0090b0 tx=0x7f661c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661c010840 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6620111d10 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f661c010e70 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f661c024e40 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6620112880 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f661c0109a0 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.738+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f662005f5f0 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.739+0000 7f66167fc700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f660c036830 0x7f660c038ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.739+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f661c04a970 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.739+0000 7f66256c0700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f660c036830 0x7f660c038ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.739+0000 7f66256c0700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f660c036830 0x7f660c038ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6610009990 tx=0x7f6610006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:55.743+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f661c02e440 con 0x7f6620102770 2024-03-24T11:10:56.586 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.033+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f662004bed0 con 0x7f6620102770 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.035+0000 7f66167fc700 1 -- 172.21.15.100:0/615150739 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f661c02c340 con 0x7f6620102770 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f660c036830 msgr2=0x7f660c038ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f660c036830 0x7f660c038ce0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f6610009990 tx=0x7f6610006e40 comp rx=0 tx=0).stop 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 msgr2=0x7f66201115d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f66201115d0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7f661c0090b0 tx=0x7f661c009250 comp rx=0 tx=0).stop 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 shutdown_connections 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f660c036830 0x7f660c038ce0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 --2- 172.21.15.100:0/615150739 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6620102770 0x7f66201115d0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.037+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 >> 172.21.15.100:0/615150739 conn(0x7f66200fde90 msgr2=0x7f66200feac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.038+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 shutdown_connections 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr 2024-03-24T11:10:56.038+0000 7f6628125700 1 -- 172.21.15.100:0/615150739 wait complete. 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.587 INFO:teuthology.orchestra.run.smithi100.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout:Or, if you are only running a single cluster on this host: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout:Please consider enabling telemetry to help improve Ceph: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: ceph telemetry on 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout:For more information see: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:10:56.588 INFO:teuthology.orchestra.run.smithi100.stdout:Bootstrap complete. 2024-03-24T11:10:56.619 INFO:tasks.cephadm:Fetching config... 2024-03-24T11:10:56.619 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:10:56.619 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-03-24T11:10:56.675 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-03-24T11:10:56.675 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:10:56.675 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-03-24T11:10:56.730 INFO:tasks.cephadm:Fetching mon keyring... 2024-03-24T11:10:56.730 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:10:56.730 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/keyring of=/dev/stdout 2024-03-24T11:10:56.799 INFO:tasks.cephadm:Fetching pub ssh key... 2024-03-24T11:10:56.799 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:10:56.799 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-03-24T11:10:56.855 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-03-24T11:10:56.855 DEBUG:teuthology.orchestra.run.smithi100:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1DXidWUrKTiQNp7+fb65m4J88vicxNAPrNHILyvoYa7gk7IPsv5bkNYAwKSP1kA98/uv/YoxPJW69CC2gZkRoLyt/QlznuUtPTiAy5zg+b9QsTcGWYNgSk3ByAGctz/+dHOrIgYU8EYg7x29g7nfxtuDHdfw9WO60dKyIbFOBc85MvImuO4NpyB3rjunPV3JOBhcDw3M7avYw6g5GNWLijZtKLluM5sjA2/yan0vGwObOxbtdzJMNP77bDafEoCq0+HQeZ5XJY4ueuCGrRfGCKmwQ5q1K7fvrx8iRmc/6fDF4LTCcOEmDMzCljmYbJXHcG1gvPXWumm9g8gX+S0019iHtjPDRedO9lDMi+3Vr3i/VYjLWOXbo8zfqfLy0/55hk/+I6+n3PXUt2LoTu/FFKq7O4tjMz9PdzODpCVgSpu+cqDeouXpa2J0mUXwrX/ozx1012i/kqtF1TOObEojTtHbNOMZQekwkjPBtLZKS7ysigtNDJ9B6nR/SQBxmP6U= ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:10:56.947 INFO:teuthology.orchestra.run.smithi100.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1DXidWUrKTiQNp7+fb65m4J88vicxNAPrNHILyvoYa7gk7IPsv5bkNYAwKSP1kA98/uv/YoxPJW69CC2gZkRoLyt/QlznuUtPTiAy5zg+b9QsTcGWYNgSk3ByAGctz/+dHOrIgYU8EYg7x29g7nfxtuDHdfw9WO60dKyIbFOBc85MvImuO4NpyB3rjunPV3JOBhcDw3M7avYw6g5GNWLijZtKLluM5sjA2/yan0vGwObOxbtdzJMNP77bDafEoCq0+HQeZ5XJY4ueuCGrRfGCKmwQ5q1K7fvrx8iRmc/6fDF4LTCcOEmDMzCljmYbJXHcG1gvPXWumm9g8gX+S0019iHtjPDRedO9lDMi+3Vr3i/VYjLWOXbo8zfqfLy0/55hk/+I6+n3PXUt2LoTu/FFKq7O4tjMz9PdzODpCVgSpu+cqDeouXpa2J0mUXwrX/ozx1012i/kqtF1TOObEojTtHbNOMZQekwkjPBtLZKS7ysigtNDJ9B6nR/SQBxmP6U= ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:10:56.961 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1DXidWUrKTiQNp7+fb65m4J88vicxNAPrNHILyvoYa7gk7IPsv5bkNYAwKSP1kA98/uv/YoxPJW69CC2gZkRoLyt/QlznuUtPTiAy5zg+b9QsTcGWYNgSk3ByAGctz/+dHOrIgYU8EYg7x29g7nfxtuDHdfw9WO60dKyIbFOBc85MvImuO4NpyB3rjunPV3JOBhcDw3M7avYw6g5GNWLijZtKLluM5sjA2/yan0vGwObOxbtdzJMNP77bDafEoCq0+HQeZ5XJY4ueuCGrRfGCKmwQ5q1K7fvrx8iRmc/6fDF4LTCcOEmDMzCljmYbJXHcG1gvPXWumm9g8gX+S0019iHtjPDRedO9lDMi+3Vr3i/VYjLWOXbo8zfqfLy0/55hk/+I6+n3PXUt2LoTu/FFKq7O4tjMz9PdzODpCVgSpu+cqDeouXpa2J0mUXwrX/ozx1012i/kqtF1TOObEojTtHbNOMZQekwkjPBtLZKS7ysigtNDJ9B6nR/SQBxmP6U= ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-03-24T11:10:57.008 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC1DXidWUrKTiQNp7+fb65m4J88vicxNAPrNHILyvoYa7gk7IPsv5bkNYAwKSP1kA98/uv/YoxPJW69CC2gZkRoLyt/QlznuUtPTiAy5zg+b9QsTcGWYNgSk3ByAGctz/+dHOrIgYU8EYg7x29g7nfxtuDHdfw9WO60dKyIbFOBc85MvImuO4NpyB3rjunPV3JOBhcDw3M7avYw6g5GNWLijZtKLluM5sjA2/yan0vGwObOxbtdzJMNP77bDafEoCq0+HQeZ5XJY4ueuCGrRfGCKmwQ5q1K7fvrx8iRmc/6fDF4LTCcOEmDMzCljmYbJXHcG1gvPXWumm9g8gX+S0019iHtjPDRedO9lDMi+3Vr3i/VYjLWOXbo8zfqfLy0/55hk/+I6+n3PXUt2LoTu/FFKq7O4tjMz9PdzODpCVgSpu+cqDeouXpa2J0mUXwrX/ozx1012i/kqtF1TOObEojTtHbNOMZQekwkjPBtLZKS7ysigtNDJ9B6nR/SQBxmP6U= ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:10:57.023 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-03-24T11:10:57.277 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:10:57.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:10:57 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/615150739' entity='client.admin' 2024-03-24T11:11:00.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 -- 172.21.15.100:0/4207564166 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 msgr2=0x7f54580fe4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:00.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 --2- 172.21.15.100:0/4207564166 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f54580fe4e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7f5450009c60 tx=0x7f5450004d10 comp rx=0 tx=0).stop 2024-03-24T11:11:00.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 -- 172.21.15.100:0/4207564166 shutdown_connections 2024-03-24T11:11:00.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 --2- 172.21.15.100:0/4207564166 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f54580fe4e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:00.040 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 -- 172.21.15.100:0/4207564166 >> 172.21.15.100:0/4207564166 conn(0x7f54580f6870 msgr2=0x7f54580f8ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 -- 172.21.15.100:0/4207564166 shutdown_connections 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 -- 172.21.15.100:0/4207564166 wait complete. 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.038+0000 7f545d513700 1 Processor -- start 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f545d513700 1 -- start start 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f545d513700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f545810d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f545d513700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f545810d620 con 0x7f54580fe0d0 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f5456ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f545810d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f5456ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f545810d0e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:38252/0 (socket says 172.21.15.100:38252) 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f5456ffd700 1 -- 172.21.15.100:0/2613532948 learned_addr learned my addr 172.21.15.100:0/2613532948 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f5456ffd700 1 -- 172.21.15.100:0/2613532948 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5450009940 con 0x7f54580fe0d0 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.039+0000 7f5456ffd700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f545810d0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5450009500 tx=0x7f5450004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.040+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5450007c80 con 0x7f54580fe0d0 2024-03-24T11:11:00.041 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.040+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f545001d910 con 0x7f54580fe0d0 2024-03-24T11:11:00.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.040+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f545810d820 con 0x7f54580fe0d0 2024-03-24T11:11:00.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.040+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f545810e390 con 0x7f54580fe0d0 2024-03-24T11:11:00.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.040+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54500129d0 con 0x7f54580fe0d0 2024-03-24T11:11:00.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.041+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5450019040 con 0x7f54580fe0d0 2024-03-24T11:11:00.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.041+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54380052e0 con 0x7f54580fe0d0 2024-03-24T11:11:00.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.041+0000 7f544f7fe700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5444036880 0x7f5444038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:00.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.041+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f545001da80 con 0x7f54580fe0d0 2024-03-24T11:11:00.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.041+0000 7f54567fc700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5444036880 0x7f5444038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:00.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.042+0000 7f54567fc700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5444036880 0x7f5444038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5440009990 tx=0x7f5440006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:00.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.046+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5450026400 con 0x7f54580fe0d0 2024-03-24T11:11:00.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.242+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f5438005b10 con 0x7f54580fe0d0 2024-03-24T11:11:00.248 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.248+0000 7f544f7fe700 1 -- 172.21.15.100:0/2613532948 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7f5450017020 con 0x7f54580fe0d0 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.250+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5444036880 msgr2=0x7f5444038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.250+0000 7f545d513700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5444036880 0x7f5444038d30 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7f5440009990 tx=0x7f5440006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.250+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 msgr2=0x7f545810d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.250+0000 7f545d513700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f545810d0e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f5450009500 tx=0x7f5450004d10 comp rx=0 tx=0).stop 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.251+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 shutdown_connections 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.251+0000 7f545d513700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5444036880 0x7f5444038d30 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.251+0000 7f545d513700 1 --2- 172.21.15.100:0/2613532948 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f54580fe0d0 0x7f545810d0e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.251+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 >> 172.21.15.100:0/2613532948 conn(0x7f54580f6870 msgr2=0x7f5458103bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:00.251 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.251+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 shutdown_connections 2024-03-24T11:11:00.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:00.251+0000 7f545d513700 1 -- 172.21.15.100:0/2613532948 wait complete. 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi100", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi100", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi100", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:11:00.293 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:00 smithi100 ceph-mon[28941]: Deploying daemon crash.smithi100 on smithi100 2024-03-24T11:11:00.916 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-03-24T11:11:00.916 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-03-24T11:11:01.511 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:01 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2613532948' entity='client.admin' 2024-03-24T11:11:01.605 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:11:04.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 -- 172.21.15.100:0/3485298353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 msgr2=0x7f68400fd450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 --2- 172.21.15.100:0/3485298353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400fd450 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f68300048c0 tx=0x7f6830009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 -- 172.21.15.100:0/3485298353 shutdown_connections 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 --2- 172.21.15.100:0/3485298353 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400fd450 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 -- 172.21.15.100:0/3485298353 >> 172.21.15.100:0/3485298353 conn(0x7f68400f70c0 msgr2=0x7f68400f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 -- 172.21.15.100:0/3485298353 shutdown_connections 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.130+0000 7f684628e700 1 -- 172.21.15.100:0/3485298353 wait complete. 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f684628e700 1 Processor -- start 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f684628e700 1 -- start start 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f684628e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400f9050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f684628e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68400f9590 con 0x7f68400fd040 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f683ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400f9050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:04.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f683ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400f9050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:38256/0 (socket says 172.21.15.100:38256) 2024-03-24T11:11:04.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.131+0000 7f683ffff700 1 -- 172.21.15.100:0/3120665279 learned_addr learned my addr 172.21.15.100:0/3120665279 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:11:04.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.132+0000 7f683ffff700 1 -- 172.21.15.100:0/3120665279 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6830005040 con 0x7f68400fd040 2024-03-24T11:11:04.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.132+0000 7f683ffff700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400f9050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f68300096b0 tx=0x7f6830009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:04.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.132+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f683000bed0 con 0x7f68400fd040 2024-03-24T11:11:04.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.132+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f683000b5c0 con 0x7f68400fd040 2024-03-24T11:11:04.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.132+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68400f7660 con 0x7f68400fd040 2024-03-24T11:11:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.132+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68400f7a90 con 0x7f68400fd040 2024-03-24T11:11:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.133+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f683001c5b0 con 0x7f68400fd040 2024-03-24T11:11:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.133+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f683001c710 con 0x7f68400fd040 2024-03-24T11:11:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.133+0000 7f683cff9700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68280368d0 0x7f6828038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.133+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6830013a80 con 0x7f68400fd040 2024-03-24T11:11:04.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.133+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f684005f5f0 con 0x7f68400fd040 2024-03-24T11:11:04.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.134+0000 7f683f7fe700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68280368d0 0x7f6828038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:04.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.134+0000 7f683f7fe700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68280368d0 0x7f6828038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6834009990 tx=0x7f6834006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:04.139 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.139+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6830025400 con 0x7f68400fd040 2024-03-24T11:11:04.331 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.331+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7f68400f7f40 con 0x7f68280368d0 2024-03-24T11:11:04.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.334+0000 7f683cff9700 1 -- 172.21.15.100:0/3120665279 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f68400f7f40 con 0x7f68280368d0 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68280368d0 msgr2=0x7f6828038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68280368d0 0x7f6828038d80 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f6834009990 tx=0x7f6834006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 msgr2=0x7f68400f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400f9050 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f68300096b0 tx=0x7f6830009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 shutdown_connections 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68280368d0 0x7f6828038d80 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 --2- 172.21.15.100:0/3120665279 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f68400fd040 0x7f68400f9050 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 >> 172.21.15.100:0/3120665279 conn(0x7f68400f70c0 msgr2=0x7f6840065900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 shutdown_connections 2024-03-24T11:11:04.337 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:04.336+0000 7f684628e700 1 -- 172.21.15.100:0/3120665279 wait complete. 2024-03-24T11:11:04.916 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi105 2024-03-24T11:11:04.916 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:11:04.916 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.conf 2024-03-24T11:11:04.934 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:11:04.934 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:11:04.990 INFO:tasks.cephadm:Adding host smithi105 to orchestrator... 2024-03-24T11:11:04.990 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch host add smithi105 2024-03-24T11:11:05.258 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:11:05.381 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:11:05.382 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:05.382 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:05.382 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:05.382 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:05.382 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:05.382 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:05 smithi100 ceph-mon[28941]: Deploying daemon node-exporter.smithi100 on smithi100 2024-03-24T11:11:06.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 -- 172.21.15.100:0/540985776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 msgr2=0x7f56a00fce90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 --2- 172.21.15.100:0/540985776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a00fce90 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f56900048c0 tx=0x7f5690009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 -- 172.21.15.100:0/540985776 shutdown_connections 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 --2- 172.21.15.100:0/540985776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a00fce90 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 -- 172.21.15.100:0/540985776 >> 172.21.15.100:0/540985776 conn(0x7f56a00f8220 msgr2=0x7f56a00fa630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 -- 172.21.15.100:0/540985776 shutdown_connections 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 -- 172.21.15.100:0/540985776 wait complete. 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.777+0000 7f56a672f700 1 Processor -- start 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f56a672f700 1 -- start start 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f56a672f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a0071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f56a672f700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56a00722a0 con 0x7f56a00fca80 2024-03-24T11:11:06.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a0071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a0071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:38296/0 (socket says 172.21.15.100:38296) 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569ffff700 1 -- 172.21.15.100:0/2495795140 learned_addr learned my addr 172.21.15.100:0/2495795140 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569ffff700 1 -- 172.21.15.100:0/2495795140 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5690005040 con 0x7f56a00fca80 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569ffff700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a0071d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5690009680 tx=0x7f5690009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f569000b5c0 con 0x7f56a00fca80 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f569000be70 con 0x7f56a00fca80 2024-03-24T11:11:06.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f56a0070370 con 0x7f56a00fca80 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.778+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f56900135a0 con 0x7f56a00fca80 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.779+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56a0070860 con 0x7f56a00fca80 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.779+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5690013810 con 0x7f56a00fca80 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.779+0000 7f569cff9700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f56880368d0 0x7f5688038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.779+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5690049360 con 0x7f56a00fca80 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.780+0000 7f569f7fe700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f56880368d0 0x7f5688038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:06.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.780+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56a005f5f0 con 0x7f56a00fca80 2024-03-24T11:11:06.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.780+0000 7f569f7fe700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f56880368d0 0x7f5688038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5694009990 tx=0x7f5694006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.785+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5690021020 con 0x7f56a00fca80 2024-03-24T11:11:06.995 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:06.994+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}) v1 -- 0x7f56a0070aa0 con 0x7f56880368d0 2024-03-24T11:11:07.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:07 smithi100 ceph-mon[28941]: from='client.14188 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:11:09.129 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:08 smithi100 ceph-mon[28941]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:09.129 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:08 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:09.130 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:08 smithi100 ceph-mon[28941]: Deploying cephadm binary to smithi105 2024-03-24T11:11:09.130 INFO:teuthology.orchestra.run.smithi100.stdout:Added host 'smithi105' with addr '172.21.15.105' 2024-03-24T11:11:09.130 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.128+0000 7f569cff9700 1 -- 172.21.15.100:0/2495795140 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f56a0070aa0 con 0x7f56880368d0 2024-03-24T11:11:09.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f56880368d0 msgr2=0x7f5688038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:09.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f56880368d0 0x7f5688038d80 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f5694009990 tx=0x7f5694006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:09.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 msgr2=0x7f56a0071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:09.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a0071d60 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7f5690009680 tx=0x7f5690009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:09.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 shutdown_connections 2024-03-24T11:11:09.131 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f56880368d0 0x7f5688038d80 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:09.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 --2- 172.21.15.100:0/2495795140 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f56a00fca80 0x7f56a0071d60 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:09.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 >> 172.21.15.100:0/2495795140 conn(0x7f56a00f8220 msgr2=0x7f56a0103bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:09.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 shutdown_connections 2024-03-24T11:11:09.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:09.131+0000 7f56a672f700 1 -- 172.21.15.100:0/2495795140 wait complete. 2024-03-24T11:11:10.282 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:10 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:10.283 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:10 smithi100 ceph-mon[28941]: Added host smithi105 2024-03-24T11:11:10.305 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch host ls --format=json 2024-03-24T11:11:10.794 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:11:11.221 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:11 smithi100 ceph-mon[28941]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:11.221 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:11 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:11.221 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:11 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:11.221 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:11 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:11.221 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:11 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:12.375 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:12 smithi100 ceph-mon[28941]: Deploying daemon alertmanager.smithi100 on smithi100 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.374+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/711602756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 msgr2=0x7f8cb80fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.374+0000 7f8cbe8a0700 1 --2- 172.21.15.100:0/711602756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb80fce80 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f8ca80048c0 tx=0x7f8ca8009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/711602756 shutdown_connections 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 --2- 172.21.15.100:0/711602756 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb80fce80 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/711602756 >> 172.21.15.100:0/711602756 conn(0x7f8cb80f81d0 msgr2=0x7f8cb80fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/711602756 shutdown_connections 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/711602756 wait complete. 2024-03-24T11:11:12.427 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 Processor -- start 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 -- start start 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb8071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.375+0000 7f8cbe8a0700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cb80722a0 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb8071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb8071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:59088/0 (socket says 172.21.15.100:59088) 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb7fff700 1 -- 172.21.15.100:0/4115091262 learned_addr learned my addr 172.21.15.100:0/4115091262 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb7fff700 1 -- 172.21.15.100:0/4115091262 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca8005040 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb7fff700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb8071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009680 tx=0x7f8ca8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ca800b5c0 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8ca800be70 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cb8070370 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ca80135a0 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.376+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cb8070860 con 0x7f8cb80fca70 2024-03-24T11:11:12.428 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.377+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f8ca8013860 con 0x7f8cb80fca70 2024-03-24T11:11:12.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.377+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cb805f5f0 con 0x7f8cb80fca70 2024-03-24T11:11:12.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.377+0000 7f8cb4ff9700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8ca0036880 0x7f8ca0038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:12.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.377+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8ca80493a0 con 0x7f8cb80fca70 2024-03-24T11:11:12.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.377+0000 7f8cb77fe700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8ca0036880 0x7f8ca0038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:12.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.378+0000 7f8cb77fe700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8ca0036880 0x7f8ca0038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009990 tx=0x7f8cac006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:12.429 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.382+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8ca8011960 con 0x7f8cb80fca70 2024-03-24T11:11:12.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.581+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f8cb81064c0 con 0x7f8ca0036880 2024-03-24T11:11:12.582 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.582+0000 7f8cb4ff9700 1 -- 172.21.15.100:0/4115091262 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f8cb81064c0 con 0x7f8ca0036880 2024-03-24T11:11:12.583 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:11:12.583 INFO:teuthology.orchestra.run.smithi100.stdout:[{"addr": "172.21.15.100", "hostname": "smithi100", "labels": [], "status": ""}, {"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}] 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.584+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8ca0036880 msgr2=0x7f8ca0038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.584+0000 7f8cbe8a0700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8ca0036880 0x7f8ca0038d30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f8cac009990 tx=0x7f8cac006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.584+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 msgr2=0x7f8cb8071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.584+0000 7f8cbe8a0700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb8071d60 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f8ca8009680 tx=0x7f8ca8009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.584+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 shutdown_connections 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.584+0000 7f8cbe8a0700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8ca0036880 0x7f8ca0038d30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.585+0000 7f8cbe8a0700 1 --2- 172.21.15.100:0/4115091262 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8cb80fca70 0x7f8cb8071d60 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.585+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 >> 172.21.15.100:0/4115091262 conn(0x7f8cb80f81d0 msgr2=0x7f8cb8103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:12.585 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.585+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 shutdown_connections 2024-03-24T11:11:12.586 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:12.585+0000 7f8cbe8a0700 1 -- 172.21.15.100:0/4115091262 wait complete. 2024-03-24T11:11:13.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:13 smithi100 ceph-mon[28941]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:13 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:13.630 INFO:tasks.cephadm:Setting crush tunables to default 2024-03-24T11:11:13.631 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd crush tunables default 2024-03-24T11:11:14.084 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:11:14.478 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:14 smithi100 ceph-mon[28941]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:11:15.416 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:15 smithi100 ceph-mon[28941]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 -- 172.21.15.100:0/12654855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 msgr2=0x7f55700fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 --2- 172.21.15.100:0/12654855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f55700fce80 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f55600048c0 tx=0x7f5560009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 -- 172.21.15.100:0/12654855 shutdown_connections 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 --2- 172.21.15.100:0/12654855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f55700fce80 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 -- 172.21.15.100:0/12654855 >> 172.21.15.100:0/12654855 conn(0x7f55700f81d0 msgr2=0x7f55700fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 -- 172.21.15.100:0/12654855 shutdown_connections 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 -- 172.21.15.100:0/12654855 wait complete. 2024-03-24T11:11:15.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.785+0000 7f5576670700 1 Processor -- start 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f5576670700 1 -- start start 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f5576670700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f557010d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f5576670700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f557010d600 con 0x7f55700fca70 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f557010d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556ffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f557010d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:59096/0 (socket says 172.21.15.100:59096) 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556ffff700 1 -- 172.21.15.100:0/4169550289 learned_addr learned my addr 172.21.15.100:0/4169550289 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556ffff700 1 -- 172.21.15.100:0/4169550289 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5560005040 con 0x7f55700fca70 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556ffff700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f557010d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5560009680 tx=0x7f5560009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f556000b5c0 con 0x7f55700fca70 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f556000be70 con 0x7f55700fca70 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.786+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f557010d800 con 0x7f55700fca70 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.787+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f55600135a0 con 0x7f55700fca70 2024-03-24T11:11:15.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.787+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f557010e370 con 0x7f55700fca70 2024-03-24T11:11:15.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.788+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5570102f80 con 0x7f55700fca70 2024-03-24T11:11:15.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.788+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f556001ca90 con 0x7f55700fca70 2024-03-24T11:11:15.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.788+0000 7f556cff9700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f55580368d0 0x7f5558038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:15.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.788+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5560049350 con 0x7f55700fca70 2024-03-24T11:11:15.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.788+0000 7f556f7fe700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f55580368d0 0x7f5558038d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:15.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.788+0000 7f556f7fe700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f55580368d0 0x7f5558038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5564009990 tx=0x7f5564006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:15.793 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.793+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5560011940 con 0x7f55700fca70 2024-03-24T11:11:15.996 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:15.995+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f557010ed80 con 0x7f55700fca70 2024-03-24T11:11:16.151 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.151+0000 7f556cff9700 1 -- 172.21.15.100:0/4169550289 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5560021020 con 0x7f55700fca70 2024-03-24T11:11:16.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f55580368d0 msgr2=0x7f5558038d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:16.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f55580368d0 0x7f5558038d80 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f5564009990 tx=0x7f5564006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:16.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 msgr2=0x7f557010d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:16.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f557010d0c0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f5560009680 tx=0x7f5560009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:16.153 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 shutdown_connections 2024-03-24T11:11:16.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f55580368d0 0x7f5558038d80 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:16.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 --2- 172.21.15.100:0/4169550289 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f55700fca70 0x7f557010d0c0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:16.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 >> 172.21.15.100:0/4169550289 conn(0x7f55700f81d0 msgr2=0x7f5570103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:16.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 shutdown_connections 2024-03-24T11:11:16.154 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:11:16.153+0000 7f5576670700 1 -- 172.21.15.100:0/4169550289 wait complete. 2024-03-24T11:11:16.154 INFO:teuthology.orchestra.run.smithi100.stderr:adjusted tunables profile to default 2024-03-24T11:11:16.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:16 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/4169550289' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:11:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:17 smithi100 ceph-mon[28941]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:17 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/4169550289' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:11:17.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:17 smithi100 ceph-mon[28941]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:11:17.896 INFO:tasks.cephadm:Adding mon.smithi100 on smithi100 2024-03-24T11:11:17.896 INFO:tasks.cephadm:Adding mon.smithi105 on smithi105 2024-03-24T11:11:17.897 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch apply mon '2;smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105' 2024-03-24T11:11:18.145 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:18.192 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:19.294 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:19 smithi100 ceph-mon[28941]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.796+0000 7f4a1c157700 1 -- 172.21.15.105:0/2765453693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 msgr2=0x7f4a140fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.796+0000 7f4a1c157700 1 --2- 172.21.15.105:0/2765453693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a140fce60 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f4a100048c0 tx=0x7f4a10009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.796+0000 7f4a1c157700 1 -- 172.21.15.105:0/2765453693 shutdown_connections 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.796+0000 7f4a1c157700 1 --2- 172.21.15.105:0/2765453693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a140fce60 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.796+0000 7f4a1c157700 1 -- 172.21.15.105:0/2765453693 >> 172.21.15.105:0/2765453693 conn(0x7f4a140f8190 msgr2=0x7f4a140fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a1c157700 1 -- 172.21.15.105:0/2765453693 shutdown_connections 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a1c157700 1 -- 172.21.15.105:0/2765453693 wait complete. 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a1c157700 1 Processor -- start 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a1c157700 1 -- start start 2024-03-24T11:11:19.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a1c157700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a14071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a1c157700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a140722a0 con 0x7f4a140fca50 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a19ef3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a14071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a19ef3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a14071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:46124/0 (socket says 172.21.15.105:46124) 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.797+0000 7f4a19ef3700 1 -- 172.21.15.105:0/2440046075 learned_addr learned my addr 172.21.15.105:0/2440046075 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a19ef3700 1 -- 172.21.15.105:0/2440046075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a10005040 con 0x7f4a140fca50 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a19ef3700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a14071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a100096b0 tx=0x7f4a10009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1000bed0 con 0x7f4a140fca50 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a14070370 con 0x7f4a140fca50 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4a1000b5c0 con 0x7f4a140fca50 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a1001c5b0 con 0x7f4a140fca50 2024-03-24T11:11:19.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.798+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a14070860 con 0x7f4a140fca50 2024-03-24T11:11:19.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.799+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f4a10013a80 con 0x7f4a140fca50 2024-03-24T11:11:19.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.799+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a1405f5f0 con 0x7f4a140fca50 2024-03-24T11:11:19.801 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.799+0000 7f4a0a7fc700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4a00036880 0x7f4a00038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:19.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.799+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4a10049ae0 con 0x7f4a140fca50 2024-03-24T11:11:19.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.799+0000 7f4a196f2700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4a00036880 0x7f4a00038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:19.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.800+0000 7f4a196f2700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4a00036880 0x7f4a00038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009990 tx=0x7f4a04006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:19.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.804+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4a10011940 con 0x7f4a140fca50 2024-03-24T11:11:19.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:19.997+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 --> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105", "target": ["mon-mgr", ""]}) v1 -- 0x7f4a14106550 con 0x7f4a00036880 2024-03-24T11:11:20.004 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.003+0000 7f4a0a7fc700 1 -- 172.21.15.105:0/2440046075 <== mgr.14162 v2:172.21.15.100:6800/1493018813 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f4a14106550 con 0x7f4a00036880 2024-03-24T11:11:20.004 INFO:teuthology.orchestra.run.smithi105.stdout:Scheduled mon update... 2024-03-24T11:11:20.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4a00036880 msgr2=0x7f4a00038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:20.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4a00036880 0x7f4a00038d30 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f4a04009990 tx=0x7f4a04006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:20.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 msgr2=0x7f4a14071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:20.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a14071d60 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4a100096b0 tx=0x7f4a10009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:20.006 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 shutdown_connections 2024-03-24T11:11:20.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4a00036880 0x7f4a00038d30 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:20.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 --2- 172.21.15.105:0/2440046075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4a140fca50 0x7f4a14071d60 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:20.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 >> 172.21.15.105:0/2440046075 conn(0x7f4a140f8190 msgr2=0x7f4a14103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:20.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 shutdown_connections 2024-03-24T11:11:20.007 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:20.005+0000 7f4a1c157700 1 -- 172.21.15.105:0/2440046075 wait complete. 2024-03-24T11:11:20.559 DEBUG:teuthology.orchestra.run.smithi105:mon.smithi105> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi105.service 2024-03-24T11:11:20.561 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:20.562 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:20.838 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:20.884 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:21.350 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:21 smithi100 ceph-mon[28941]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:21.350 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:21 smithi100 ceph-mon[28941]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:11:21.350 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:21 smithi100 ceph-mon[28941]: Saving service mon spec with placement smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105;count:2 2024-03-24T11:11:21.351 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:21 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 -- 172.21.15.105:0/4275905193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 msgr2=0x7f9a440fce70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 --2- 172.21.15.105:0/4275905193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a440fce70 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f9a380048c0 tx=0x7f9a38009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 -- 172.21.15.105:0/4275905193 shutdown_connections 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 --2- 172.21.15.105:0/4275905193 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a440fce70 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 -- 172.21.15.105:0/4275905193 >> 172.21.15.105:0/4275905193 conn(0x7f9a440f81c0 msgr2=0x7f9a440fa5f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 -- 172.21.15.105:0/4275905193 shutdown_connections 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.306+0000 7f9a494e5700 1 -- 172.21.15.105:0/4275905193 wait complete. 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a494e5700 1 Processor -- start 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a494e5700 1 -- start start 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a494e5700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a44071d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:22.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a494e5700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a440722a0 con 0x7f9a440fca60 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a42ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a44071d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a42ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a44071d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:46136/0 (socket says 172.21.15.105:46136) 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.307+0000 7f9a42ffd700 1 -- 172.21.15.105:0/4219626850 learned_addr learned my addr 172.21.15.105:0/4219626850 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a42ffd700 1 -- 172.21.15.105:0/4219626850 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a38005040 con 0x7f9a440fca60 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a42ffd700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a44071d60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a380096b0 tx=0x7f9a38009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a3800bed0 con 0x7f9a440fca60 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f9a3800b5c0 con 0x7f9a440fca60 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a44070370 con 0x7f9a440fca60 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a3801c5b0 con 0x7f9a440fca60 2024-03-24T11:11:22.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.308+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a44070860 con 0x7f9a440fca60 2024-03-24T11:11:22.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.309+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f9a3801c820 con 0x7f9a440fca60 2024-03-24T11:11:22.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.309+0000 7f9a2b7fe700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f9a2c036880 0x7f9a2c038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:22.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.309+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9a38049b30 con 0x7f9a440fca60 2024-03-24T11:11:22.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.309+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a4405f5f0 con 0x7f9a440fca60 2024-03-24T11:11:22.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.309+0000 7f9a427fc700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f9a2c036880 0x7f9a2c038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:22.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.309+0000 7f9a427fc700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f9a2c036880 0x7f9a2c038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009990 tx=0x7f9a34006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:22.314 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.314+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9a38023300 con 0x7f9a440fca60 2024-03-24T11:11:22.576 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.576+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a44106440 con 0x7f9a440fca60 2024-03-24T11:11:22.577 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.576+0000 7f9a2b7fe700 1 -- 172.21.15.105:0/4219626850 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9a38011b60 con 0x7f9a440fca60 2024-03-24T11:11:22.577 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:22.577 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f9a2c036880 msgr2=0x7f9a2c038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f9a2c036880 0x7f9a2c038d30 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f9a34009990 tx=0x7f9a34006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 msgr2=0x7f9a44071d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a44071d60 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f9a380096b0 tx=0x7f9a38009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 shutdown_connections 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f9a2c036880 0x7f9a2c038d30 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:22.579 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 --2- 172.21.15.105:0/4219626850 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9a440fca60 0x7f9a44071d60 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:22.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 >> 172.21.15.105:0/4219626850 conn(0x7f9a440f81c0 msgr2=0x7f9a44103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:22.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 shutdown_connections 2024-03-24T11:11:22.580 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:22.578+0000 7f9a494e5700 1 -- 172.21.15.105:0/4219626850 wait complete. 2024-03-24T11:11:22.580 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:22.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: Deploying daemon grafana.smithi100 on smithi100 2024-03-24T11:11:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:22 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/4219626850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:24.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:23 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:24.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:24.172 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:24.423 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:24.469 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:25.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:24 smithi100 ceph-mon[28941]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 -- 172.21.15.105:0/2277239942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 msgr2=0x7f8d140fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 --2- 172.21.15.105:0/2277239942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d140fd460 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f8d080048c0 tx=0x7f8d08009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 -- 172.21.15.105:0/2277239942 shutdown_connections 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 --2- 172.21.15.105:0/2277239942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d140fd460 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 -- 172.21.15.105:0/2277239942 >> 172.21.15.105:0/2277239942 conn(0x7f8d140f70d0 msgr2=0x7f8d140f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 -- 172.21.15.105:0/2277239942 shutdown_connections 2024-03-24T11:11:25.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 -- 172.21.15.105:0/2277239942 wait complete. 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.791+0000 7f8d1aece700 1 Processor -- start 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d1aece700 1 -- start start 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d1aece700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d14065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d1aece700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d1410fde0 con 0x7f8d140fd050 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d18c6a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d14065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d18c6a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d14065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:39592/0 (socket says 172.21.15.105:39592) 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d18c6a700 1 -- 172.21.15.105:0/1099155735 learned_addr learned my addr 172.21.15.105:0/1099155735 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.792+0000 7f8d18c6a700 1 -- 172.21.15.105:0/1099155735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d08005040 con 0x7f8d140fd050 2024-03-24T11:11:25.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d18c6a700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d14065bd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d080096b0 tx=0x7f8d08009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:25.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d0800bed0 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8d0800b5c0 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d0801c5b0 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d1410ff20 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d14110410 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.793+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f8d0801c820 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.794+0000 7f8d117fa700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8d04036880 0x7f8d04038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.794+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f8d08049ad0 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.794+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d14105920 con 0x7f8d140fd050 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.794+0000 7f8d13fff700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8d04036880 0x7f8d04038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:25.796 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.794+0000 7f8d13fff700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8d04036880 0x7f8d04038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009990 tx=0x7f8d00006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:25.800 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:25.799+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8d08023300 con 0x7f8d140fd050 2024-03-24T11:11:26.062 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.061+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d14101e70 con 0x7f8d140fd050 2024-03-24T11:11:26.063 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.062+0000 7f8d117fa700 1 -- 172.21.15.105:0/1099155735 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f8d08034090 con 0x7f8d140fd050 2024-03-24T11:11:26.063 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:26.063 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8d04036880 msgr2=0x7f8d04038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8d04036880 0x7f8d04038d30 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f8d00009990 tx=0x7f8d00006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 msgr2=0x7f8d14065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d14065bd0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f8d080096b0 tx=0x7f8d08009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 shutdown_connections 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f8d04036880 0x7f8d04038d30 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 --2- 172.21.15.105:0/1099155735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8d140fd050 0x7f8d14065bd0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 >> 172.21.15.105:0/1099155735 conn(0x7f8d140f70d0 msgr2=0x7f8d14102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 shutdown_connections 2024-03-24T11:11:26.065 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:26.064+0000 7f8d1aece700 1 -- 172.21.15.105:0/1099155735 wait complete. 2024-03-24T11:11:26.066 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:27.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:26 smithi100 ceph-mon[28941]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:27.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:26 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/1099155735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:27.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:27.635 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:27.883 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:27.930 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:29.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:28 smithi100 ceph-mon[28941]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 -- 172.21.15.105:0/2568208767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 msgr2=0x7f5dec0fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 --2- 172.21.15.105:0/2568208767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec0fc1f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f5de00048c0 tx=0x7f5de0009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 -- 172.21.15.105:0/2568208767 shutdown_connections 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 --2- 172.21.15.105:0/2568208767 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec0fc1f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 -- 172.21.15.105:0/2568208767 >> 172.21.15.105:0/2568208767 conn(0x7f5dec0f6850 msgr2=0x7f5dec0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 -- 172.21.15.105:0/2568208767 shutdown_connections 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 -- 172.21.15.105:0/2568208767 wait complete. 2024-03-24T11:11:29.395 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 Processor -- start 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.394+0000 7f5df26c6700 1 -- start start 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5df26c6700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec1136c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5df26c6700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dec113c00 con 0x7f5dec0fbde0 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5debfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec1136c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5debfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec1136c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:39612/0 (socket says 172.21.15.105:39612) 2024-03-24T11:11:29.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5debfff700 1 -- 172.21.15.105:0/1687664776 learned_addr learned my addr 172.21.15.105:0/1687664776 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5debfff700 1 -- 172.21.15.105:0/1687664776 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5de0005040 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.395+0000 7f5debfff700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec1136c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5de00096b0 tx=0x7f5de0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5de000bed0 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dec113e00 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5de000b5c0 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5de001c5b0 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dec114970 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f5de001c820 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.396+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dec100d50 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.397+0000 7f5de8ff9700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5dd4036880 0x7f5dd4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.397+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5de0049b00 con 0x7f5dec0fbde0 2024-03-24T11:11:29.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.397+0000 7f5deb7fe700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5dd4036880 0x7f5dd4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:29.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.397+0000 7f5deb7fe700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5dd4036880 0x7f5dd4038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009990 tx=0x7f5ddc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:29.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.402+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5de0023300 con 0x7f5dec0fbde0 2024-03-24T11:11:29.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.661+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5dec115380 con 0x7f5dec0fbde0 2024-03-24T11:11:29.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.662+0000 7f5de8ff9700 1 -- 172.21.15.105:0/1687664776 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5de0011b60 con 0x7f5dec0fbde0 2024-03-24T11:11:29.663 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:29.663 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5dd4036880 msgr2=0x7f5dd4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5dd4036880 0x7f5dd4038d30 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f5ddc009990 tx=0x7f5ddc006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 msgr2=0x7f5dec1136c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec1136c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f5de00096b0 tx=0x7f5de0009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 shutdown_connections 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5dd4036880 0x7f5dd4038d30 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 --2- 172.21.15.105:0/1687664776 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5dec0fbde0 0x7f5dec1136c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 >> 172.21.15.105:0/1687664776 conn(0x7f5dec0f6850 msgr2=0x7f5dec10aa80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 shutdown_connections 2024-03-24T11:11:29.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:29.664+0000 7f5df26c6700 1 -- 172.21.15.105:0/1687664776 wait complete. 2024-03-24T11:11:29.666 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:30.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:29 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/1687664776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:31.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:30 smithi100 ceph-mon[28941]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:31.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:31.261 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:31.509 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:31.555 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:32.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:32 smithi100 ceph-mon[28941]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:32.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 -- 172.21.15.105:0/369702235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 msgr2=0x7f484c0fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:32.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 --2- 172.21.15.105:0/369702235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c0fce60 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f48400048c0 tx=0x7f4840009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:32.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 -- 172.21.15.105:0/369702235 shutdown_connections 2024-03-24T11:11:32.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 --2- 172.21.15.105:0/369702235 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c0fce60 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:32.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 -- 172.21.15.105:0/369702235 >> 172.21.15.105:0/369702235 conn(0x7f484c0f8190 msgr2=0x7f484c0fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:32.954 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 -- 172.21.15.105:0/369702235 shutdown_connections 2024-03-24T11:11:32.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.952+0000 7f4851732700 1 -- 172.21.15.105:0/369702235 wait complete. 2024-03-24T11:11:32.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.953+0000 7f4851732700 1 Processor -- start 2024-03-24T11:11:32.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.953+0000 7f4851732700 1 -- start start 2024-03-24T11:11:32.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.953+0000 7f4851732700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c10d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:32.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.953+0000 7f4851732700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f484c10d5f0 con 0x7f484c0fca50 2024-03-24T11:11:32.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f484affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c10d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:32.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f484affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c10d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:39624/0 (socket says 172.21.15.105:39624) 2024-03-24T11:11:32.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f484affd700 1 -- 172.21.15.105:0/3468985556 learned_addr learned my addr 172.21.15.105:0/3468985556 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:32.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f484affd700 1 -- 172.21.15.105:0/3468985556 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4840005040 con 0x7f484c0fca50 2024-03-24T11:11:32.956 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f484affd700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c10d0b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48400096b0 tx=0x7f4840009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f484000bed0 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f484000b5c0 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f484c10d7f0 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f484001c5b0 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.954+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f484c10e360 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.955+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f4840013420 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.955+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f484c102f60 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.955+0000 7f48337fe700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4834036880 0x7f4834038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.955+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4840049a50 con 0x7f484c0fca50 2024-03-24T11:11:32.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.956+0000 7f484a7fc700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4834036880 0x7f4834038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:32.958 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.956+0000 7f484a7fc700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4834036880 0x7f4834038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f483c009990 tx=0x7f483c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:32.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:32.960+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4840025400 con 0x7f484c0fca50 2024-03-24T11:11:33.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.219+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f484c10ed70 con 0x7f484c0fca50 2024-03-24T11:11:33.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.220+0000 7f48337fe700 1 -- 172.21.15.105:0/3468985556 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f4840004d30 con 0x7f484c0fca50 2024-03-24T11:11:33.221 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4834036880 msgr2=0x7f4834038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4834036880 0x7f4834038d30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f483c009990 tx=0x7f483c006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 msgr2=0x7f484c10d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c10d0b0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f48400096b0 tx=0x7f4840009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 shutdown_connections 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f4834036880 0x7f4834038d30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 --2- 172.21.15.105:0/3468985556 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f484c0fca50 0x7f484c10d0b0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 >> 172.21.15.105:0/3468985556 conn(0x7f484c0f8190 msgr2=0x7f484c103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 shutdown_connections 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:33.222+0000 7f4851732700 1 -- 172.21.15.105:0/3468985556 wait complete. 2024-03-24T11:11:33.223 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:33.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:33 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/3468985556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:34.731 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:34.731 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:34.981 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:35.028 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:35.224 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:35.224 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:35.224 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:35.224 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:35.224 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:35.225 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:35.225 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:35.225 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:35 smithi100 ceph-mon[28941]: Deploying daemon prometheus.smithi100 on smithi100 2024-03-24T11:11:36.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 -- 172.21.15.105:0/3336535809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 msgr2=0x7f09940fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 --2- 172.21.15.105:0/3336535809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f09940fd460 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7f09840048c0 tx=0x7f0984009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 -- 172.21.15.105:0/3336535809 shutdown_connections 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 --2- 172.21.15.105:0/3336535809 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f09940fd460 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 -- 172.21.15.105:0/3336535809 >> 172.21.15.105:0/3336535809 conn(0x7f09940f70d0 msgr2=0x7f09940f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 -- 172.21.15.105:0/3336535809 shutdown_connections 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.597+0000 7f099c6d7700 1 -- 172.21.15.105:0/3336535809 wait complete. 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099c6d7700 1 Processor -- start 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099c6d7700 1 -- start start 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099c6d7700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f099410bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:36.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099c6d7700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f099410c500 con 0x7f09940fd050 2024-03-24T11:11:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099a473700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f099410bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099a473700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f099410bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:49444/0 (socket says 172.21.15.105:49444) 2024-03-24T11:11:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.598+0000 7f099a473700 1 -- 172.21.15.105:0/4096211205 learned_addr learned my addr 172.21.15.105:0/4096211205 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f099a473700 1 -- 172.21.15.105:0/4096211205 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0984005040 con 0x7f09940fd050 2024-03-24T11:11:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f099a473700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f099410bfc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f09840096b0 tx=0x7f0984009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:36.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f098400bed0 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f098400b5c0 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f099410c700 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f098401c5b0 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.599+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f099410d270 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.600+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f0984013420 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.600+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0994101e70 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.600+0000 7f098affd700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0980036880 0x7f0980038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.600+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f0984049b10 con 0x7f09940fd050 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.600+0000 7f0999c72700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0980036880 0x7f0980038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:36.602 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.600+0000 7f0999c72700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0980036880 0x7f0980038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0990009990 tx=0x7f0990006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:36.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.605+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0984025400 con 0x7f09940fd050 2024-03-24T11:11:36.867 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.866+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f099410dc80 con 0x7f09940fd050 2024-03-24T11:11:36.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.867+0000 7f098affd700 1 -- 172.21.15.105:0/4096211205 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f0984004d30 con 0x7f09940fd050 2024-03-24T11:11:36.868 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:36.868 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0980036880 msgr2=0x7f0980038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0980036880 0x7f0980038d30 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f0990009990 tx=0x7f0990006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 msgr2=0x7f099410bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f099410bfc0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f09840096b0 tx=0x7f0984009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 shutdown_connections 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f0980036880 0x7f0980038d30 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 --2- 172.21.15.105:0/4096211205 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09940fd050 0x7f099410bfc0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 >> 172.21.15.105:0/4096211205 conn(0x7f09940f70d0 msgr2=0x7f0994102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 shutdown_connections 2024-03-24T11:11:36.870 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:36.869+0000 7f099c6d7700 1 -- 172.21.15.105:0/4096211205 wait complete. 2024-03-24T11:11:36.871 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:37.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:37 smithi100 ceph-mon[28941]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:37.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:37 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/4096211205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:38.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:38.452 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:38.705 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:38.757 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:39.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:38 smithi100 ceph-mon[28941]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:39.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:38 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:40.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 -- 172.21.15.105:0/2540636905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 msgr2=0x7f690c0ff6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:40.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 --2- 172.21.15.105:0/2540636905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c0ff6f0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f69080048c0 tx=0x7f6908009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:40.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 -- 172.21.15.105:0/2540636905 shutdown_connections 2024-03-24T11:11:40.240 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 --2- 172.21.15.105:0/2540636905 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c0ff6f0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:40.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 -- 172.21.15.105:0/2540636905 >> 172.21.15.105:0/2540636905 conn(0x7f690c0f70c0 msgr2=0x7f690c0f94f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:40.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 -- 172.21.15.105:0/2540636905 shutdown_connections 2024-03-24T11:11:40.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.239+0000 7f6913109700 1 -- 172.21.15.105:0/2540636905 wait complete. 2024-03-24T11:11:40.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6913109700 1 Processor -- start 2024-03-24T11:11:40.241 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6913109700 1 -- start start 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6913109700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c10d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6913109700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c10d600 con 0x7f690c066940 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6910ea5700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c10d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6910ea5700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c10d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:49466/0 (socket says 172.21.15.105:49466) 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6910ea5700 1 -- 172.21.15.105:0/2035063734 learned_addr learned my addr 172.21.15.105:0/2035063734 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.240+0000 7f6910ea5700 1 -- 172.21.15.105:0/2035063734 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6908005040 con 0x7f690c066940 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.241+0000 7f6910ea5700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c10d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f690800b880 tx=0x7f6908009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.241+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690800bbc0 con 0x7f690c066940 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.241+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f690800bd20 con 0x7f690c066940 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.241+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c10d800 con 0x7f690c066940 2024-03-24T11:11:40.243 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.241+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f690801c690 con 0x7f690c066940 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.241+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c10e370 con 0x7f690c066940 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.242+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f690801c8d0 con 0x7f690c066940 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.242+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c102f70 con 0x7f690c066940 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.242+0000 7f69017fa700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68f4036880 0x7f68f4038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.242+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6908049cb0 con 0x7f690c066940 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.242+0000 7f6903fff700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68f4036880 0x7f68f4038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:40.244 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.242+0000 7f6903fff700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68f4036880 0x7f68f4038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009990 tx=0x7f68fc006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:40.247 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.247+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6908023300 con 0x7f690c066940 2024-03-24T11:11:40.506 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.506+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f690c10ed80 con 0x7f690c066940 2024-03-24T11:11:40.507 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.506+0000 7f69017fa700 1 -- 172.21.15.105:0/2035063734 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6908011b60 con 0x7f690c066940 2024-03-24T11:11:40.507 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:40.507 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:40.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68f4036880 msgr2=0x7f68f4038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:40.509 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68f4036880 0x7f68f4038d30 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f68fc009990 tx=0x7f68fc006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 msgr2=0x7f690c10d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c10d0c0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f690800b880 tx=0x7f6908009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 shutdown_connections 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f68f4036880 0x7f68f4038d30 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 --2- 172.21.15.105:0/2035063734 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f690c066940 0x7f690c10d0c0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 >> 172.21.15.105:0/2035063734 conn(0x7f690c0f70c0 msgr2=0x7f690c103ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 shutdown_connections 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:40.509+0000 7f6913109700 1 -- 172.21.15.105:0/2035063734 wait complete. 2024-03-24T11:11:40.510 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:41.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:40 smithi100 ceph-mon[28941]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:41.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:40 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/2035063734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:42.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:42.071 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:42.318 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:42.366 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:43.136 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:42 smithi100 ceph-mon[28941]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 -- 172.21.15.105:0/3218686741 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 msgr2=0x7f6d440fce60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 --2- 172.21.15.105:0/3218686741 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d440fce60 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009c60 tx=0x7f6d34004d10 comp rx=0 tx=0).stop 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 -- 172.21.15.105:0/3218686741 shutdown_connections 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 --2- 172.21.15.105:0/3218686741 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d440fce60 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 -- 172.21.15.105:0/3218686741 >> 172.21.15.105:0/3218686741 conn(0x7f6d440f8190 msgr2=0x7f6d440fa5e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 -- 172.21.15.105:0/3218686741 shutdown_connections 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 -- 172.21.15.105:0/3218686741 wait complete. 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.813+0000 7f6d495ab700 1 Processor -- start 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d495ab700 1 -- start start 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d495ab700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d4410d0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d495ab700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d4410d5f0 con 0x7f6d440fca50 2024-03-24T11:11:43.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d42ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d4410d0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:43.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d42ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d4410d0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:51392/0 (socket says 172.21.15.105:51392) 2024-03-24T11:11:43.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d42ffd700 1 -- 172.21.15.105:0/3675140169 learned_addr learned my addr 172.21.15.105:0/3675140169 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:43.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d42ffd700 1 -- 172.21.15.105:0/3675140169 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d34009940 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.814+0000 7f6d42ffd700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d4410d0b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009500 tx=0x7f6d34004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.815+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d34007c80 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.815+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d4410d7f0 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.815+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6d3401d910 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.815+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6d340129d0 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.815+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d4410e360 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.816+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d44102f60 con 0x7f6d440fca50 2024-03-24T11:11:43.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.816+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 13) v1 ==== 43425+0+0 (secure 0 0 0) 0x7f6d34019040 con 0x7f6d440fca50 2024-03-24T11:11:43.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.816+0000 7f6d3bfff700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f6d30036880 0x7f6d30038d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:43.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.816+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6d3404a390 con 0x7f6d440fca50 2024-03-24T11:11:43.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.816+0000 7f6d3a7fc700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f6d30036880 0x7f6d30038d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:43.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.817+0000 7f6d3a7fc700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f6d30036880 0x7f6d30038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009990 tx=0x7f6d2c006e40 comp rx=0 tx=0).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:43.821+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6d3402eb40 con 0x7f6d440fca50 2024-03-24T11:11:44.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.082+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6d4410ecc0 con 0x7f6d440fca50 2024-03-24T11:11:44.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.083+0000 7f6d3bfff700 1 -- 172.21.15.105:0/3675140169 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6d34017020 con 0x7f6d440fca50 2024-03-24T11:11:44.084 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:44.084 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:44.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f6d30036880 msgr2=0x7f6d30038d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:44.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f6d30036880 0x7f6d30038d30 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7f6d2c009990 tx=0x7f6d2c006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:44.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 msgr2=0x7f6d4410d0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:44.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d4410d0b0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f6d34009500 tx=0x7f6d34004d10 comp rx=0 tx=0).stop 2024-03-24T11:11:44.085 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 shutdown_connections 2024-03-24T11:11:44.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f6d30036880 0x7f6d30038d30 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:44.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 --2- 172.21.15.105:0/3675140169 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6d440fca50 0x7f6d4410d0b0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:44.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 >> 172.21.15.105:0/3675140169 conn(0x7f6d440f8190 msgr2=0x7f6d44103b90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:44.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 shutdown_connections 2024-03-24T11:11:44.086 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:44.085+0000 7f6d495ab700 1 -- 172.21.15.105:0/3675140169 wait complete. 2024-03-24T11:11:44.086 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:44.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:44 smithi100 ceph-mon[28941]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:44.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:44 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/3675140169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:45.632 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:45.632 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:45.871 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:45.919 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:46.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:46 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:46.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:46 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:46.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:46 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:46.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:46 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:11:46.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:46 smithi100 ceph-mon[28941]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:11:47.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 -- 172.21.15.105:0/1775420337 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 msgr2=0x7f5d7c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:47.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 --2- 172.21.15.105:0/1775420337 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c0fd460 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5d700048c0 tx=0x7f5d70009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:47.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 -- 172.21.15.105:0/1775420337 shutdown_connections 2024-03-24T11:11:47.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 --2- 172.21.15.105:0/1775420337 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c0fd460 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:47.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 -- 172.21.15.105:0/1775420337 >> 172.21.15.105:0/1775420337 conn(0x7f5d7c0f70d0 msgr2=0x7f5d7c0f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:47.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 -- 172.21.15.105:0/1775420337 shutdown_connections 2024-03-24T11:11:47.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.256+0000 7f5d824c6700 1 -- 172.21.15.105:0/1775420337 wait complete. 2024-03-24T11:11:47.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.257+0000 7f5d824c6700 1 Processor -- start 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.257+0000 7f5d824c6700 1 -- start start 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.257+0000 7f5d824c6700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c10bfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.257+0000 7f5d824c6700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c10c500 con 0x7f5d7c0fd050 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.257+0000 7f5d7bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c10bfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d7bfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c10bfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:51410/0 (socket says 172.21.15.105:51410) 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d7bfff700 1 -- 172.21.15.105:0/1033502747 learned_addr learned my addr 172.21.15.105:0/1033502747 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d7bfff700 1 -- 172.21.15.105:0/1033502747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d70005040 con 0x7f5d7c0fd050 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d7bfff700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c10bfc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009680 tx=0x7f5d70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:47.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d7000b5c0 con 0x7f5d7c0fd050 2024-03-24T11:11:47.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5d7000be70 con 0x7f5d7c0fd050 2024-03-24T11:11:47.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c10c700 con 0x7f5d7c0fd050 2024-03-24T11:11:47.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d700135a0 con 0x7f5d7c0fd050 2024-03-24T11:11:47.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.258+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c10d270 con 0x7f5d7c0fd050 2024-03-24T11:11:47.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.259+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f5d7001ca90 con 0x7f5d7c0fd050 2024-03-24T11:11:47.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.259+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c101e70 con 0x7f5d7c0fd050 2024-03-24T11:11:47.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.259+0000 7f5d78ff9700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 0x7f5d64038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:47.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.259+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f5d700494a0 con 0x7f5d7c0fd050 2024-03-24T11:11:47.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.259+0000 7f5d7b7fe700 1 -- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 msgr2=0x7f5d64038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1493018813 2024-03-24T11:11:47.261 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.259+0000 7f5d7b7fe700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 0x7f5d64038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:11:47.264 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.265+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5d70011960 con 0x7f5d7c0fd050 2024-03-24T11:11:47.460 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.460+0000 7f5d7b7fe700 1 -- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 msgr2=0x7f5d64038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1493018813 2024-03-24T11:11:47.461 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.460+0000 7f5d7b7fe700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 0x7f5d64038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T11:11:47.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.523+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d7c10dc80 con 0x7f5d7c0fd050 2024-03-24T11:11:47.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.523+0000 7f5d78ff9700 1 -- 172.21.15.105:0/1033502747 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5d70021070 con 0x7f5d7c0fd050 2024-03-24T11:11:47.524 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:47.524 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:47.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 msgr2=0x7f5d64038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:11:47.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 0x7f5d64038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 msgr2=0x7f5d7c10bfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c10bfc0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f5d70009680 tx=0x7f5d70009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 shutdown_connections 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f5d640368d0 0x7f5d64038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 --2- 172.21.15.105:0/1033502747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5d7c0fd050 0x7f5d7c10bfc0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 >> 172.21.15.105:0/1033502747 conn(0x7f5d7c0f70d0 msgr2=0x7f5d7c102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 shutdown_connections 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:47.526+0000 7f5d824c6700 1 -- 172.21.15.105:0/1033502747 wait complete. 2024-03-24T11:11:47.527 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:47.976 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:47 smithi100 ceph-mon[28941]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:11:47.976 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:47 smithi100 ceph-mon[28941]: mgrmap e14: smithi100.smxgyi(active, since 59s) 2024-03-24T11:11:47.976 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:47 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/1033502747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:49.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:49.088 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:49.340 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:49.386 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:50.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 -- 172.21.15.105:0/3940926174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 msgr2=0x7f01ec0fce80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:50.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 --2- 172.21.15.105:0/3940926174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec0fce80 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f01e00048c0 tx=0x7f01e0009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:50.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 -- 172.21.15.105:0/3940926174 shutdown_connections 2024-03-24T11:11:50.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 --2- 172.21.15.105:0/3940926174 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec0fce80 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:50.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 -- 172.21.15.105:0/3940926174 >> 172.21.15.105:0/3940926174 conn(0x7f01ec0f81d0 msgr2=0x7f01ec0fa600 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:50.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 -- 172.21.15.105:0/3940926174 shutdown_connections 2024-03-24T11:11:50.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.725+0000 7f01f2e9d700 1 -- 172.21.15.105:0/3940926174 wait complete. 2024-03-24T11:11:50.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f2e9d700 1 Processor -- start 2024-03-24T11:11:50.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f2e9d700 1 -- start start 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f2e9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec10d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f2e9d700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01ec10d600 con 0x7f01ec0fca70 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f0c39700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec10d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f0c39700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec10d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:51422/0 (socket says 172.21.15.105:51422) 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.726+0000 7f01f0c39700 1 -- 172.21.15.105:0/778300527 learned_addr learned my addr 172.21.15.105:0/778300527 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01f0c39700 1 -- 172.21.15.105:0/778300527 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e0005040 con 0x7f01ec0fca70 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01f0c39700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec10d0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f01e00096b0 tx=0x7f01e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e000bed0 con 0x7f01ec0fca70 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01e000b5c0 con 0x7f01ec0fca70 2024-03-24T11:11:50.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01ec10d800 con 0x7f01ec0fca70 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e001c5b0 con 0x7f01ec0fca70 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.727+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01ec10e370 con 0x7f01ec0fca70 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.728+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 14) v1 ==== 43439+0+0 (secure 0 0 0) 0x7f01e0013420 con 0x7f01ec0fca70 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.728+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01ec102f80 con 0x7f01ec0fca70 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.728+0000 7f01e97fa700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 0x7f01dc038d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.728+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f01e0049b10 con 0x7f01ec0fca70 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.728+0000 7f01ebfff700 1 -- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 msgr2=0x7f01dc038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1493018813 2024-03-24T11:11:50.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.728+0000 7f01ebfff700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 0x7f01dc038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:11:50.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.733+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f01e0013a80 con 0x7f01ec0fca70 2024-03-24T11:11:50.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.929+0000 7f01ebfff700 1 -- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 msgr2=0x7f01dc038d80 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.100:6800/1493018813 2024-03-24T11:11:50.929 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.929+0000 7f01ebfff700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 0x7f01dc038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-03-24T11:11:50.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.991+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01ec10ed80 con 0x7f01ec0fca70 2024-03-24T11:11:50.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.992+0000 7f01e97fa700 1 -- 172.21.15.105:0/778300527 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f01e0011370 con 0x7f01ec0fca70 2024-03-24T11:11:50.992 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:50.992 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 msgr2=0x7f01dc038d80 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 0x7f01dc038d80 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 msgr2=0x7f01ec10d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec10d0c0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f01e00096b0 tx=0x7f01e0009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 shutdown_connections 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:6800/1493018813,v1:172.21.15.100:6801/1493018813] conn(0x7f01dc0368d0 0x7f01dc038d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 --2- 172.21.15.105:0/778300527 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01ec0fca70 0x7f01ec10d0c0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.993+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 >> 172.21.15.105:0/778300527 conn(0x7f01ec0f81d0 msgr2=0x7f01ec103bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.994+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 shutdown_connections 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:50.994+0000 7f01f2e9d700 1 -- 172.21.15.105:0/778300527 wait complete. 2024-03-24T11:11:50.994 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:51.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:51 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/778300527' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:52.499 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:52.499 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:52.752 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:52.799 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: Active manager daemon smithi100.smxgyi restarted 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: Activating manager daemon smithi100.smxgyi 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: mgrmap e15: smithi100.smxgyi(active, starting, since 0.00303082s) 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi100.smxgyi", "id": "smithi100.smxgyi"}]: dispatch 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:11:53.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:11:53.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:11:53.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: Manager daemon smithi100.smxgyi is now available 2024-03-24T11:11:53.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:11:53.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:11:54.325 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 -- 172.21.15.105:0/2892001349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 msgr2=0x7f7ab00fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 --2- 172.21.15.105:0/2892001349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab00fd460 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8009c60 tx=0x7f7aa8004d10 comp rx=0 tx=0).stop 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 -- 172.21.15.105:0/2892001349 shutdown_connections 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 --2- 172.21.15.105:0/2892001349 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab00fd460 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 -- 172.21.15.105:0/2892001349 >> 172.21.15.105:0/2892001349 conn(0x7f7ab00f70d0 msgr2=0x7f7ab00f9500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 -- 172.21.15.105:0/2892001349 shutdown_connections 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.313+0000 7f7ab4de3700 1 -- 172.21.15.105:0/2892001349 wait complete. 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7ab4de3700 1 Processor -- start 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7ab4de3700 1 -- start start 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7ab4de3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab0065bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7ab4de3700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ab010fde0 con 0x7f7ab00fd050 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7aae59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab0065bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7aae59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab0065bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:33666/0 (socket says 172.21.15.105:33666) 2024-03-24T11:11:54.326 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7aae59c700 1 -- 172.21.15.105:0/3411073961 learned_addr learned my addr 172.21.15.105:0/3411073961 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.314+0000 7f7aae59c700 1 -- 172.21.15.105:0/3411073961 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7aa8009940 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.315+0000 7f7aae59c700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab0065bd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8004e60 tx=0x7f7aa8004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.315+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7aa8012410 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.315+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7aa8012a40 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.315+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ab010ff20 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.315+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7aa801ba40 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.315+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ab0110410 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.316+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 16) v1 ==== 43289+0+0 (secure 0 0 0) 0x7f7aa8019040 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.316+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ab0105920 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.316+0000 7f7aa6ffd700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7a9c036710 0x7f7a9c038bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.316+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f7aa8049b60 con 0x7f7ab00fd050 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.316+0000 7f7aadd9b700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7a9c036710 0x7f7a9c038bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.316+0000 7f7aadd9b700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7a9c036710 0x7f7a9c038bc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009990 tx=0x7f7a98006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:54.327 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.322+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7aa8012570 con 0x7f7ab00fd050 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/trash_purge_schedule"}]: dispatch 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: [24/Mar/2024:11:11:53] ENGINE Bus STARTING 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: [24/Mar/2024:11:11:53] ENGINE Serving on https://172.21.15.100:7150 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: [24/Mar/2024:11:11:53] ENGINE Bus STARTED 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:54.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:54 smithi100 ceph-mon[28941]: mgrmap e16: smithi100.smxgyi(active, since 1.00788s) 2024-03-24T11:11:54.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.598+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7ab005f5f0 con 0x7f7ab00fd050 2024-03-24T11:11:54.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.598+0000 7f7aa6ffd700 1 -- 172.21.15.105:0/3411073961 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7aa8017090 con 0x7f7ab00fd050 2024-03-24T11:11:54.598 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:54.598 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:54.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.600+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7a9c036710 msgr2=0x7f7a9c038bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7a9c036710 0x7f7a9c038bc0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f7a98009990 tx=0x7f7a98006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 msgr2=0x7f7ab0065bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab0065bd0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7f7aa8004e60 tx=0x7f7aa8004d10 comp rx=0 tx=0).stop 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 shutdown_connections 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7a9c036710 0x7f7a9c038bc0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 --2- 172.21.15.105:0/3411073961 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7ab00fd050 0x7f7ab0065bd0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 >> 172.21.15.105:0/3411073961 conn(0x7f7ab00f70d0 msgr2=0x7f7ab0102aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 shutdown_connections 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:54.601+0000 7f7ab4de3700 1 -- 172.21.15.105:0/3411073961 wait complete. 2024-03-24T11:11:54.601 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:11:55.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:55.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:55 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/3411073961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:11:55.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:56.208 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:11:56.209 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:11:56.485 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:56.701 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:56 smithi100 ceph-mon[28941]: mgrmap e17: smithi100.smxgyi(active, since 2s) 2024-03-24T11:11:56.875 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /etc/ceph/ceph.conf 2024-03-24T11:11:58.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi100", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:11:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 -- 172.21.15.105:0/1347089920 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 msgr2=0x7f09f00fdeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 --2- 172.21.15.105:0/1347089920 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f00fdeb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f09e00048c0 tx=0x7f09e0009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 -- 172.21.15.105:0/1347089920 shutdown_connections 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 --2- 172.21.15.105:0/1347089920 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f00fdeb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 -- 172.21.15.105:0/1347089920 >> 172.21.15.105:0/1347089920 conn(0x7f09f0071800 msgr2=0x7f09f0071c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 -- 172.21.15.105:0/1347089920 shutdown_connections 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 -- 172.21.15.105:0/1347089920 wait complete. 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.146+0000 7f09f63dd700 1 Processor -- start 2024-03-24T11:11:59.148 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09f63dd700 1 -- start start 2024-03-24T11:11:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09f63dd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f01131c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09f63dd700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09f0113700 con 0x7f09f00fdae0 2024-03-24T11:11:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09effff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f01131c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09effff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f01131c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:33682/0 (socket says 172.21.15.105:33682) 2024-03-24T11:11:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09effff700 1 -- 172.21.15.105:0/1488639328 learned_addr learned my addr 172.21.15.105:0/1488639328 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:11:59.149 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.147+0000 7f09effff700 1 -- 172.21.15.105:0/1488639328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09e0005040 con 0x7f09f00fdae0 2024-03-24T11:11:59.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.148+0000 7f09effff700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f01131c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09e000b7f0 tx=0x7f09e0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:59.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.148+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e00092e0 con 0x7f09f00fdae0 2024-03-24T11:11:59.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.148+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f09e0009910 con 0x7f09f00fdae0 2024-03-24T11:11:59.150 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.148+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f09e001c400 con 0x7f09f00fdae0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.148+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09f0113900 con 0x7f09f00fdae0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.148+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09f0113d10 con 0x7f09f00fdae0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.149+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f09e001c640 con 0x7f09f00fdae0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.149+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09f00658f0 con 0x7f09f00fdae0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.149+0000 7f09ecff9700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f09d80369b0 0x7f09d8038e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.149+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f09e0049ef0 con 0x7f09f00fdae0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.149+0000 7f09ef7fe700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f09d80369b0 0x7f09d8038e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:11:59.151 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.149+0000 7f09ef7fe700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f09d80369b0 0x7f09d8038e60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009990 tx=0x7f09e4006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:11:59.154 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.154+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f09e001b7d0 con 0x7f09f00fdae0 2024-03-24T11:11:59.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:59 smithi100 ceph-mon[28941]: Updating smithi100:/etc/ceph/ceph.conf 2024-03-24T11:11:59.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:59 smithi100 ceph-mon[28941]: Updating smithi105:/etc/ceph/ceph.conf 2024-03-24T11:11:59.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:59 smithi100 ceph-mon[28941]: Updating smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:11:59.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:59 smithi100 ceph-mon[28941]: Updating smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:11:59.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:59 smithi100 ceph-mon[28941]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:11:59.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:11:59 smithi100 ceph-mon[28941]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:11:59.436 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.435+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09f005f5f0 con 0x7f09f00fdae0 2024-03-24T11:11:59.437 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.436+0000 7f09ecff9700 1 -- 172.21.15.105:0/1488639328 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f09e001b7d0 con 0x7f09f00fdae0 2024-03-24T11:11:59.437 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:11:59.437 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f09d80369b0 msgr2=0x7f09d8038e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f09d80369b0 0x7f09d8038e60 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f09e4009990 tx=0x7f09e4006e40 comp rx=0 tx=0).stop 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 msgr2=0x7f09f01131c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f01131c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f09e000b7f0 tx=0x7f09e0009e30 comp rx=0 tx=0).stop 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 shutdown_connections 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f09d80369b0 0x7f09d8038e60 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:59.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 --2- 172.21.15.105:0/1488639328 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f09f00fdae0 0x7f09f01131c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:11:59.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 >> 172.21.15.105:0/1488639328 conn(0x7f09f0071800 msgr2=0x7f09f0103730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:11:59.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 shutdown_connections 2024-03-24T11:11:59.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:11:59.439+0000 7f09f63dd700 1 -- 172.21.15.105:0/1488639328 wait complete. 2024-03-24T11:11:59.441 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: Updating smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: Updating smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:00.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: Deploying daemon crash.smithi105 on smithi105 2024-03-24T11:12:00.645 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:00 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/1488639328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:01.032 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:12:01.032 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:12:01.326 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:03.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.354+0000 7fd5a192f700 1 -- 172.21.15.105:0/926948894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 msgr2=0x7fd59c070730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:03.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.354+0000 7fd5a192f700 1 --2- 172.21.15.105:0/926948894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c070730 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7fd58c0048c0 tx=0x7fd58c009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.354+0000 7fd5a192f700 1 -- 172.21.15.105:0/926948894 shutdown_connections 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.354+0000 7fd5a192f700 1 --2- 172.21.15.105:0/926948894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c070730 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.354+0000 7fd5a192f700 1 -- 172.21.15.105:0/926948894 >> 172.21.15.105:0/926948894 conn(0x7fd59c0f85e0 msgr2=0x7fd59c0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.355+0000 7fd5a192f700 1 -- 172.21.15.105:0/926948894 shutdown_connections 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.355+0000 7fd5a192f700 1 -- 172.21.15.105:0/926948894 wait complete. 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.355+0000 7fd5a192f700 1 Processor -- start 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.355+0000 7fd5a192f700 1 -- start start 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.355+0000 7fd5a192f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c115340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.355+0000 7fd5a192f700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c115880 con 0x7fd59c070360 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd59affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c115340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd59affd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c115340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:33710/0 (socket says 172.21.15.105:33710) 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd59affd700 1 -- 172.21.15.105:0/3193602662 learned_addr learned my addr 172.21.15.105:0/3193602662 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:12:03.361 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd59affd700 1 -- 172.21.15.105:0/3193602662 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c005040 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd59affd700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c115340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00b820 tx=0x7fd58c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c0093c0 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd58c013420 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c115a80 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58c01c5b0 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.356+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c115ef0 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.357+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7fd58c01c7f0 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.357+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c071aa0 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.358+0000 7fd5837fe700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd584036970 0x7fd584038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.358+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd58c049b30 con 0x7fd59c070360 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.358+0000 7fd59a7fc700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd584036970 0x7fd584038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:03.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.358+0000 7fd59a7fc700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd584036970 0x7fd584038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd590009990 tx=0x7fd590006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:03.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.363+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd58c023300 con 0x7fd59c070360 2024-03-24T11:12:03.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.657+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd59c05f5f0 con 0x7fd59c070360 2024-03-24T11:12:03.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.658+0000 7fd5837fe700 1 -- 172.21.15.105:0/3193602662 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd58c011b60 con 0x7fd59c070360 2024-03-24T11:12:03.659 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:12:03.659 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:12:03.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd584036970 msgr2=0x7fd584038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd584036970 0x7fd584038e20 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fd590009990 tx=0x7fd590006e40 comp rx=0 tx=0).stop 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 msgr2=0x7fd59c115340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c115340 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7fd58c00b820 tx=0x7fd58c009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 shutdown_connections 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd584036970 0x7fd584038e20 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 --2- 172.21.15.105:0/3193602662 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd59c070360 0x7fd59c115340 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 >> 172.21.15.105:0/3193602662 conn(0x7fd59c0f85e0 msgr2=0x7fd59c100d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 shutdown_connections 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:03.660+0000 7fd5a192f700 1 -- 172.21.15.105:0/3193602662 wait complete. 2024-03-24T11:12:03.661 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:12:04.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:04.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:04.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:04.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:04.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:04 smithi100 ceph-mon[28941]: Deploying daemon node-exporter.smithi105 on smithi105 2024-03-24T11:12:04.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:04 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/3193602662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:05.255 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:12:05.255 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:12:05.523 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 -- 172.21.15.105:0/1833589314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 msgr2=0x7f15c4102fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 --2- 172.21.15.105:0/1833589314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c4102fa0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f15c00048c0 tx=0x7f15c0009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 -- 172.21.15.105:0/1833589314 shutdown_connections 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 --2- 172.21.15.105:0/1833589314 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c4102fa0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 -- 172.21.15.105:0/1833589314 >> 172.21.15.105:0/1833589314 conn(0x7f15c40f85e0 msgr2=0x7f15c40fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 -- 172.21.15.105:0/1833589314 shutdown_connections 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.449+0000 7f15cc141700 1 -- 172.21.15.105:0/1833589314 wait complete. 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15cc141700 1 Processor -- start 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15cc141700 1 -- start start 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15cc141700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c40702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:08.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15cc141700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c4070800 con 0x7f15c4102bd0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15c9edd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c40702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15c9edd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c40702c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:54268/0 (socket says 172.21.15.105:54268) 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.450+0000 7f15c9edd700 1 -- 172.21.15.105:0/143197232 learned_addr learned my addr 172.21.15.105:0/143197232 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15c9edd700 1 -- 172.21.15.105:0/143197232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c0005040 con 0x7f15c4102bd0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15c9edd700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c40702c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f15c000b820 tx=0x7f15c0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15c00093c0 con 0x7f15c4102bd0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f15c0013420 con 0x7f15c4102bd0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c4070a00 con 0x7f15c4102bd0 2024-03-24T11:12:08.453 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15c001c5b0 con 0x7f15c4102bd0 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.451+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c4070ef0 con 0x7f15c4102bd0 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.452+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7f15c001c7f0 con 0x7f15c4102bd0 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.452+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c4071130 con 0x7f15c4102bd0 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.452+0000 7f15ba7fc700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f15b0036970 0x7f15b0038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.452+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f15c0049b60 con 0x7f15c4102bd0 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.452+0000 7f15c96dc700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f15b0036970 0x7f15b0038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:08.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.453+0000 7f15c96dc700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f15b0036970 0x7f15b0038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009990 tx=0x7f15b4006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:08.458 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.457+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f15c4071130 con 0x7f15c4102bd0 2024-03-24T11:12:08.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.734+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15c4071130 con 0x7f15c4102bd0 2024-03-24T11:12:08.735 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.735+0000 7f15ba7fc700 1 -- 172.21.15.105:0/143197232 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f15c4071130 con 0x7f15c4102bd0 2024-03-24T11:12:08.736 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:12:08.736 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f15b0036970 msgr2=0x7f15b0038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f15b0036970 0x7f15b0038e20 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f15b4009990 tx=0x7f15b4006e40 comp rx=0 tx=0).stop 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 msgr2=0x7f15c40702c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c40702c0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f15c000b820 tx=0x7f15c0009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 shutdown_connections 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f15b0036970 0x7f15b0038e20 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 --2- 172.21.15.105:0/143197232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f15c4102bd0 0x7f15c40702c0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 >> 172.21.15.105:0/143197232 conn(0x7f15c40f85e0 msgr2=0x7f15c40f9cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 shutdown_connections 2024-03-24T11:12:08.738 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:08.737+0000 7f15cc141700 1 -- 172.21.15.105:0/143197232 wait complete. 2024-03-24T11:12:08.739 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: Deploying daemon mgr.smithi105.nmspmy on smithi105 2024-03-24T11:12:09.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:09.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:09.394 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:08 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/143197232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:11.086 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:12:11.086 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:11 smithi100 ceph-mon[28941]: Deploying daemon mon.smithi105 on smithi105 2024-03-24T11:12:13.123 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:12:14.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:14 smithi100 ceph-mon[28941]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:14.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:14 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:14.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1770932873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 msgr2=0x7ff0a4102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:14.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 --2- 172.21.15.105:0/1770932873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a4102020 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 crypto rx=0x7ff0980048c0 tx=0x7ff098009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:14.696 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1770932873 shutdown_connections 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 --2- 172.21.15.105:0/1770932873 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a4102020 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1770932873 >> 172.21.15.105:0/1770932873 conn(0x7ff0a40719d0 msgr2=0x7ff0a4071dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1770932873 shutdown_connections 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1770932873 wait complete. 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 Processor -- start 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.694+0000 7ff0aaf56700 1 -- start start 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0aaf56700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a410e4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0aaf56700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0a4066540 con 0x7ff0a4101c50 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0a8cf2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a410e4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0a8cf2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a410e4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.105:34484/0 (socket says 172.21.15.105:34484) 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0a8cf2700 1 -- 172.21.15.105:0/1096218145 learned_addr learned my addr 172.21.15.105:0/1096218145 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0a8cf2700 1 -- 172.21.15.105:0/1096218145 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff098005040 con 0x7ff0a4101c50 2024-03-24T11:12:14.697 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.695+0000 7ff0a8cf2700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a410e4b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff09800b820 tx=0x7ff098009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:14.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.696+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0980093f0 con 0x7ff0a4101c50 2024-03-24T11:12:14.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.696+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0a410e9f0 con 0x7ff0a4101c50 2024-03-24T11:12:14.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.696+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff098013420 con 0x7ff0a4101c50 2024-03-24T11:12:14.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.696+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff09801c570 con 0x7ff0a4101c50 2024-03-24T11:12:14.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.696+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0a410ee60 con 0x7ff0a4101c50 2024-03-24T11:12:14.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.697+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 17) v1 ==== 43483+0+0 (secure 0 0 0) 0x7ff09801c830 con 0x7ff0a4101c50 2024-03-24T11:12:14.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.697+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0a40728c0 con 0x7ff0a4101c50 2024-03-24T11:12:14.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.697+0000 7ff0a17fa700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff094036970 0x7ff094038e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:14.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.697+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7ff098049b80 con 0x7ff0a4101c50 2024-03-24T11:12:14.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.697+0000 7ff0a3fff700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff094036970 0x7ff094038e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:14.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.698+0000 7ff0a3fff700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff094036970 0x7ff094038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff090009990 tx=0x7ff090006e40 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:14.703 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.702+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff098023300 con 0x7ff0a4101c50 2024-03-24T11:12:14.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.975+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0a405f5f0 con 0x7ff0a4101c50 2024-03-24T11:12:14.976 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.975+0000 7ff0a17fa700 1 -- 172.21.15.105:0/1096218145 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7ff098011b10 con 0x7ff0a4101c50 2024-03-24T11:12:14.976 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:12:14.977 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":1,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:09:39.830576Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-03-24T11:12:14.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.977+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff094036970 msgr2=0x7ff094038e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:14.978 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.977+0000 7ff0aaf56700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff094036970 0x7ff094038e20 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff090009990 tx=0x7ff090006e40 comp rx=0 tx=0).stop 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.977+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 msgr2=0x7ff0a410e4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.977+0000 7ff0aaf56700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a410e4b0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 crypto rx=0x7ff09800b820 tx=0x7ff098009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.978+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 shutdown_connections 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.978+0000 7ff0aaf56700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff094036970 0x7ff094038e20 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.978+0000 7ff0aaf56700 1 --2- 172.21.15.105:0/1096218145 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff0a4101c50 0x7ff0a410e4b0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.978+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 >> 172.21.15.105:0/1096218145 conn(0x7ff0a40719d0 msgr2=0x7ff0a4102ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.978+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 shutdown_connections 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:14.978+0000 7ff0aaf56700 1 -- 172.21.15.105:0/1096218145 wait complete. 2024-03-24T11:12:14.979 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 1 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).mds e1 new map 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).mds e1 print_map 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: e1 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: legacy client fscid: -1 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: No filesystems configured 2024-03-24T11:12:15.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-03-24T11:12:15.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-03-24T11:12:15.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-03-24T11:12:15.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-03-24T11:12:15.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-03-24T11:12:15.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-03-24T11:12:15.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying cephadm binary to smithi105 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Added host smithi105 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon alertmanager.smithi100 on smithi100 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.14190 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/4169550289' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/4169550289' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: osdmap e4: 0 total, 0 up, 0 in 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Saving service mon spec with placement smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105;count:2 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon grafana.smithi100 on smithi100 2024-03-24T11:12:15.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/4219626850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/1099155735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/1687664776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/3468985556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.556 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon prometheus.smithi100 on smithi100 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/4096211205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/2035063734' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.557 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/3675140169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14162 172.21.15.100:0/2851182171' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mgrmap e14: smithi100.smxgyi(active, since 59s) 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/1033502747' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.558 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/778300527' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Active manager daemon smithi100.smxgyi restarted 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Activating manager daemon smithi100.smxgyi 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mgrmap e15: smithi100.smxgyi(active, starting, since 0.00303082s) 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi100.smxgyi", "id": "smithi100.smxgyi"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Manager daemon smithi100.smxgyi is now available 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.559 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/mirror_snapshot_schedule"}]: dispatch 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi100.smxgyi/trash_purge_schedule"}]: dispatch 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: [24/Mar/2024:11:11:53] ENGINE Bus STARTING 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: [24/Mar/2024:11:11:53] ENGINE Serving on https://172.21.15.100:7150 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: [24/Mar/2024:11:11:53] ENGINE Bus STARTED 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mgrmap e16: smithi100.smxgyi(active, since 1.00788s) 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/3411073961' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mgrmap e17: smithi100.smxgyi(active, since 2s) 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.560 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi100", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi100:/etc/ceph/ceph.conf 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi105:/etc/ceph/ceph.conf 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:15.561 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi100:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Updating smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.client.admin.keyring 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon crash.smithi105 on smithi105 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/1488639328' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.562 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon node-exporter.smithi105 on smithi105 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/3193602662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon mgr.smithi105.nmspmy on smithi105 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.563 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/143197232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: Deploying daemon mon.smithi105 on smithi105 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 ceph-mon[36437]: mon.smithi105@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 bash[36374]: 243836c787a46a256e0982ef262809818bad1dcbb6cd3feab77f5f2a2a80ea9d 2024-03-24T11:12:15.564 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:15 smithi105 systemd[1]: Started Ceph mon.smithi105 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:12:16.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-03-24T11:12:16.946 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mon dump -f json 2024-03-24T11:12:17.204 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:12:20.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:12:20.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: mon.smithi100 calling monitor election 2024-03-24T11:12:20.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: mon.smithi105 calling monitor election 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi105.nmspmy/crt"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: mon.smithi100 is new leader, mons smithi100,smithi105 in quorum (ranks 0,1) 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: monmap e2: 2 mons at {smithi100=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: fsmap 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: mgrmap e17: smithi100.smxgyi(active, since 27s) 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: overall HEALTH_OK 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: Standby manager daemon smithi105.nmspmy started 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi105.nmspmy/key"}]: dispatch 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: mon.smithi100 calling monitor election 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: mon.smithi105 calling monitor election 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi105.nmspmy/crt"}]: dispatch 2024-03-24T11:12:20.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: mon.smithi100 is new leader, mons smithi100,smithi105 in quorum (ranks 0,1) 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: monmap e2: 2 mons at {smithi100=[v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0],smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: fsmap 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: osdmap e5: 0 total, 0 up, 0 in 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: mgrmap e17: smithi100.smxgyi(active, since 27s) 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: overall HEALTH_OK 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: Standby manager daemon smithi105.nmspmy started 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi105.nmspmy/key"}]: dispatch 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.? 172.21.15.105:0/3923692804' entity='mgr.smithi105.nmspmy' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:20.645 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:21.476 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:21 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:21.476 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:21 smithi105 ceph-mon[36437]: mgrmap e18: smithi100.smxgyi(active, since 27s), standbys: smithi105.nmspmy 2024-03-24T11:12:21.476 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:21 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi105.nmspmy", "id": "smithi105.nmspmy"}]: dispatch 2024-03-24T11:12:21.476 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:21 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:12:21.476 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:21 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:21.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 -- 172.21.15.105:0/353989077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcdc8003580 msgr2=0x7fcdc8005a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:21.528 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 --2- 172.21.15.105:0/353989077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcdc8003580 0x7fcdc8005a00 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 crypto rx=0x7fcddc0fcae0 tx=0x7fcdd800b040 comp rx=0 tx=0).stop 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 -- 172.21.15.105:0/353989077 shutdown_connections 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 --2- 172.21.15.105:0/353989077 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcdc8003580 0x7fcdc8005a00 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 -- 172.21.15.105:0/353989077 >> 172.21.15.105:0/353989077 conn(0x7fcddc0f8730 msgr2=0x7fcddc0fab40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 -- 172.21.15.105:0/353989077 shutdown_connections 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 -- 172.21.15.105:0/353989077 wait complete. 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 Processor -- start 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.526+0000 7fcde37f3700 1 -- start start 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde37f3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde37f3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcddc113800 0x7fcddc10d220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde37f3700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc113e10 con 0x7fcddc113800 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde37f3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcddc10d760 con 0x7fcdc8003580 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:39992/0 (socket says 172.21.15.105:39992) 2024-03-24T11:12:21.529 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 -- 172.21.15.105:0/2199139660 learned_addr learned my addr 172.21.15.105:0/2199139660 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 -- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 msgr2=0x7fcddc1132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 12 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 -- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 msgr2=0x7fcddc1132c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde158f700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde0d8e700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcddc113800 0x7fcddc10d220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde0d8e700 1 -- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 msgr2=0x7fcddc1132c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde0d8e700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.527+0000 7fcde0d8e700 1 -- 172.21.15.105:0/2199139660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdd8005040 con 0x7fcddc113800 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.528+0000 7fcde0d8e700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcddc113800 0x7fcddc10d220 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00b790 tx=0x7fcdcc0091a0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.528+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdcc011810 con 0x7fcddc113800 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.528+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcddc10da40 con 0x7fcddc113800 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.528+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcdcc011e40 con 0x7fcddc113800 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.528+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcdcc00f2e0 con 0x7fcddc113800 2024-03-24T11:12:21.530 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.528+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcddc10df80 con 0x7fcddc113800 2024-03-24T11:12:21.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.529+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcddc0fe2f0 con 0x7fcddc113800 2024-03-24T11:12:21.531 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.529+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fcdcc011970 con 0x7fcddc113800 2024-03-24T11:12:21.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.530+0000 7fcdd1ffb700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcdc0069320 0x7fcdc006b7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:21.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.530+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fcdcc0863b0 con 0x7fcddc113800 2024-03-24T11:12:21.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.530+0000 7fcde158f700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcdc0069320 0x7fcdc006b7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:21.532 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.530+0000 7fcde158f700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcdc0069320 0x7fcdc006b7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007f50 tx=0x7fcdd800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:21.535 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.535+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcdcc0571d0 con 0x7fcddc113800 2024-03-24T11:12:21.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:21 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:21.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:21 smithi100 ceph-mon[28941]: mgrmap e18: smithi100.smxgyi(active, since 27s), standbys: smithi105.nmspmy 2024-03-24T11:12:21.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:21 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr metadata", "who": "smithi105.nmspmy", "id": "smithi105.nmspmy"}]: dispatch 2024-03-24T11:12:21.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:21 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:12:21.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:21 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:12:21.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.801+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcddc04bed0 con 0x7fcddc113800 2024-03-24T11:12:21.802 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.801+0000 7fcdd1ffb700 1 -- 172.21.15.105:0/2199139660 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7fcdcc015020 con 0x7fcddc113800 2024-03-24T11:12:21.802 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-03-24T11:12:21.802 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":2,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","modified":"2024-03-24T11:12:15.086280Z","created":"2024-03-24T11:09:39.830576Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi100","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:3300","nonce":0},{"type":"v1","addr":"172.21.15.100:6789","nonce":0}]},"addr":"172.21.15.100:6789/0","public_addr":"172.21.15.100:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-03-24T11:12:21.804 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcdc0069320 msgr2=0x7fcdc006b7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcdc0069320 0x7fcdc006b7d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fcdd8007f50 tx=0x7fcdd800b040 comp rx=0 tx=0).stop 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcddc113800 msgr2=0x7fcddc10d220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcddc113800 0x7fcddc10d220 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 crypto rx=0x7fcdcc00b790 tx=0x7fcdcc0091a0 comp rx=0 tx=0).stop 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 shutdown_connections 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcdc0069320 0x7fcdc006b7d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcdc8003580 0x7fcddc1132c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 --2- 172.21.15.105:0/2199139660 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fcddc113800 0x7fcddc10d220 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 >> 172.21.15.105:0/2199139660 conn(0x7fcddc0f8730 msgr2=0x7fcddc1060f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 shutdown_connections 2024-03-24T11:12:21.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:12:21.804+0000 7fcde37f3700 1 -- 172.21.15.105:0/2199139660 wait complete. 2024-03-24T11:12:21.806 INFO:teuthology.orchestra.run.smithi105.stderr:dumped monmap epoch 2 2024-03-24T11:12:22.358 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:22 smithi105 ceph-mon[36437]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:22.358 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:22 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/2199139660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:22.382 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-03-24T11:12:22.383 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph config generate-minimal-conf 2024-03-24T11:12:22.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:22 smithi100 ceph-mon[28941]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:22.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:22 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/2199139660' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-03-24T11:12:22.659 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:12:23.799 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.799 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.799 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:23.799 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:12:23.800 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: Updating smithi100:/etc/ceph/ceph.conf 2024-03-24T11:12:23.800 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: Updating smithi105:/etc/ceph/ceph.conf 2024-03-24T11:12:23.800 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.800 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.800 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.801 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.801 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.801 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:12:23.801 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.801 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:12:23.801 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:23.802 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:23.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:23.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:12:23.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: Updating smithi100:/etc/ceph/ceph.conf 2024-03-24T11:12:23.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: Updating smithi105:/etc/ceph/ceph.conf 2024-03-24T11:12:23.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:12:23.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:23.805 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:12:23.805 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:23.805 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:24.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 -- 172.21.15.100:0/3492528942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c102a70 msgr2=0x7f008c102e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:24.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 --2- 172.21.15.100:0/3492528942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c102a70 0x7f008c102e40 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7f007c0048c0 tx=0x7f007c009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:24.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 -- 172.21.15.100:0/3492528942 shutdown_connections 2024-03-24T11:12:24.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 --2- 172.21.15.100:0/3492528942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c102a70 0x7f008c102e40 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:24.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 -- 172.21.15.100:0/3492528942 >> 172.21.15.100:0/3492528942 conn(0x7f008c0f6c30 msgr2=0x7f008c0f9040 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 -- 172.21.15.100:0/3492528942 shutdown_connections 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.289+0000 7f0093c94700 1 -- 172.21.15.100:0/3492528942 wait complete. 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f0093c94700 1 Processor -- start 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f0093c94700 1 -- start start 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f0093c94700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f008c102a70 0x7f008c1132a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f0093c94700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 0x7f008c10d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f0093c94700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c113df0 con 0x7f008c1137e0 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f0093c94700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f008c10d740 con 0x7f008c102a70 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f009122f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 0x7f008c10d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f009122f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 0x7f008c10d200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:50718/0 (socket says 172.21.15.100:50718) 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f009122f700 1 -- 172.21.15.100:0/2638339463 learned_addr learned my addr 172.21.15.100:0/2638339463 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f009122f700 1 -- 172.21.15.100:0/2638339463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f008c102a70 msgr2=0x7f008c1132a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:24.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f009122f700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f008c102a70 0x7f008c1132a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.290+0000 7f009122f700 1 -- 172.21.15.100:0/2638339463 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f007c005040 con 0x7f008c1137e0 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.291+0000 7f009122f700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 0x7f008c10d200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f008800b790 tx=0x7f0088009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.291+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0088010870 con 0x7f008c1137e0 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.291+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f008800f3e0 con 0x7f008c1137e0 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.291+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0088010ea0 con 0x7f008c1137e0 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.291+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f008c10da20 con 0x7f008c1137e0 2024-03-24T11:12:24.293 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.291+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f008c10df60 con 0x7f008c1137e0 2024-03-24T11:12:24.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.292+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f008c0fe2c0 con 0x7f008c1137e0 2024-03-24T11:12:24.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.292+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f008800f550 con 0x7f008c1137e0 2024-03-24T11:12:24.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.293+0000 7f00827fc700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0078069370 0x7f007806b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:24.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.293+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f00880865e0 con 0x7f008c1137e0 2024-03-24T11:12:24.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.293+0000 7f0091a30700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0078069370 0x7f007806b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:24.294 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.293+0000 7f0091a30700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0078069370 0x7f007806b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f007c009b80 tx=0x7f007c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:24.298 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.298+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0088057480 con 0x7f008c1137e0 2024-03-24T11:12:24.486 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.486+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f008c05f5f0 con 0x7f008c1137e0 2024-03-24T11:12:24.487 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.486+0000 7f00827fc700 1 -- 172.21.15.100:0/2638339463 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7f0088014100 con 0x7f008c1137e0 2024-03-24T11:12:24.487 INFO:teuthology.orchestra.run.smithi100.stdout:# minimal ceph.conf for e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:12:24.487 INFO:teuthology.orchestra.run.smithi100.stdout:[global] 2024-03-24T11:12:24.487 INFO:teuthology.orchestra.run.smithi100.stdout: fsid = e2f9ef6c-e9ce-11ee-95cd-87774f69a715 2024-03-24T11:12:24.487 INFO:teuthology.orchestra.run.smithi100.stdout: mon_host = [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] 2024-03-24T11:12:24.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.488+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0078069370 msgr2=0x7f007806b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:24.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.488+0000 7f0093c94700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0078069370 0x7f007806b820 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f007c009b80 tx=0x7f007c009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:24.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.488+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 msgr2=0x7f008c10d200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.488+0000 7f0093c94700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 0x7f008c10d200 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7f008800b790 tx=0x7f0088009250 comp rx=0 tx=0).stop 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 shutdown_connections 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0078069370 0x7f007806b820 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f008c102a70 0x7f008c1132a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 --2- 172.21.15.100:0/2638339463 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f008c1137e0 0x7f008c10d200 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 >> 172.21.15.100:0/2638339463 conn(0x7f008c0f6c30 msgr2=0x7f008c0f7000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 shutdown_connections 2024-03-24T11:12:24.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:24.489+0000 7f0093c94700 1 -- 172.21.15.100:0/2638339463 wait complete. 2024-03-24T11:12:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:24 smithi105 ceph-mon[36437]: Updating smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:24 smithi105 ceph-mon[36437]: Updating smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:24 smithi105 ceph-mon[36437]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:24 smithi105 ceph-mon[36437]: Reconfiguring mon.smithi100 (unknown last config time)... 2024-03-24T11:12:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:24 smithi105 ceph-mon[36437]: Reconfiguring daemon mon.smithi100 on smithi100 2024-03-24T11:12:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:24 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2638339463' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:24.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:24 smithi100 ceph-mon[28941]: Updating smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:24.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:24 smithi100 ceph-mon[28941]: Updating smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/config/ceph.conf 2024-03-24T11:12:24.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:24 smithi100 ceph-mon[28941]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:24.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:24 smithi100 ceph-mon[28941]: Reconfiguring mon.smithi100 (unknown last config time)... 2024-03-24T11:12:24.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:24 smithi100 ceph-mon[28941]: Reconfiguring daemon mon.smithi100 on smithi100 2024-03-24T11:12:24.840 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:24 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2638339463' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:25.131 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-03-24T11:12:25.132 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:12:25.132 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:12:25.162 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:12:25.162 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:12:25.230 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:12:25.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-03-24T11:12:25.265 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:12:25.266 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-03-24T11:12:25.336 INFO:tasks.cephadm:Deploying OSDs... 2024-03-24T11:12:25.336 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:12:25.336 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:12:25.351 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:12:25.351 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_1 2024-03-24T11:12:25.406 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:12:25.406 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:25.406 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 665 Links: 1 2024-03-24T11:12:25.406 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:25.406 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:25.407 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-24 11:11:45.008447919 +0000 2024-03-24T11:12:25.407 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-24 11:05:53.282212158 +0000 2024-03-24T11:12:25.407 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-24 11:05:53.282212158 +0000 2024-03-24T11:12:25.407 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-24 11:05:53.282212158 +0000 2024-03-24T11:12:25.407 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:12:25.472 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-24T11:12:25.473 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-24T11:12:25.473 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000245673 s, 2.1 MB/s 2024-03-24T11:12:25.474 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:12:25.530 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_2 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 686 Links: 1 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-24 11:11:45.008447919 +0000 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-24 11:05:53.618203892 +0000 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-24 11:05:53.618203892 +0000 2024-03-24T11:12:25.586 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-24 11:05:53.618203892 +0000 2024-03-24T11:12:25.587 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:12:25.654 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-24T11:12:25.654 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-24T11:12:25.654 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000231676 s, 2.2 MB/s 2024-03-24T11:12:25.655 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:12:25.711 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_3 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 705 Links: 1 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-24 11:11:45.009447895 +0000 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-24 11:05:53.998194544 +0000 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-24 11:05:53.998194544 +0000 2024-03-24T11:12:25.766 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-24 11:05:53.998194544 +0000 2024-03-24T11:12:25.767 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:12:25.832 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-24T11:12:25.832 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-24T11:12:25.832 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000237357 s, 2.2 MB/s 2024-03-24T11:12:25.833 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:12:25.890 DEBUG:teuthology.orchestra.run.smithi100:> stat /dev/vg_nvme/lv_4 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 5h/5d Inode: 723 Links: 1 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 2024-03-24 11:11:45.009447895 +0000 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-24 11:05:54.346185982 +0000 2024-03-24T11:12:25.945 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-24 11:05:54.346185982 +0000 2024-03-24T11:12:25.946 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-24 11:05:54.346185982 +0000 2024-03-24T11:12:25.946 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:12:26.010 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records in 2024-03-24T11:12:26.011 INFO:teuthology.orchestra.run.smithi100.stderr:1+0 records out 2024-03-24T11:12:26.011 INFO:teuthology.orchestra.run.smithi100.stderr:512 bytes copied, 0.000210208 s, 2.4 MB/s 2024-03-24T11:12:26.012 DEBUG:teuthology.orchestra.run.smithi100:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:12:26.068 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:12:26.068 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-03-24T11:12:26.087 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-03-24T11:12:26.087 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-03-24T11:12:26.144 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-03-24T11:12:26.144 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 666 Links: 1 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-24 11:12:13.334698794 +0000 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-24 11:05:53.284594807 +0000 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-24 11:05:53.284594807 +0000 2024-03-24T11:12:26.145 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-24 11:05:53.284594807 +0000 2024-03-24T11:12:26.145 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-03-24T11:12:26.213 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-24T11:12:26.213 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-24T11:12:26.213 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000231214 s, 2.2 MB/s 2024-03-24T11:12:26.214 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-03-24T11:12:26.272 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 684 Links: 1 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-24 11:12:13.334698794 +0000 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-24 11:05:53.610588294 +0000 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-24 11:05:53.610588294 +0000 2024-03-24T11:12:26.329 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-24 11:05:53.610588294 +0000 2024-03-24T11:12:26.329 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-03-24T11:12:26.364 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:26 smithi100 ceph-mon[28941]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:26.394 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:26 smithi105 ceph-mon[36437]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:26.396 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-24T11:12:26.396 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-24T11:12:26.396 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000190889 s, 2.7 MB/s 2024-03-24T11:12:26.398 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-03-24T11:12:26.454 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 703 Links: 1 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-24 11:12:13.335698775 +0000 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-24 11:05:53.949581521 +0000 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-24 11:05:53.949581521 +0000 2024-03-24T11:12:26.510 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-24 11:05:53.949581521 +0000 2024-03-24T11:12:26.511 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-03-24T11:12:26.577 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-24T11:12:26.577 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-24T11:12:26.577 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000201767 s, 2.5 MB/s 2024-03-24T11:12:26.578 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-03-24T11:12:26.634 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-03-24T11:12:26.690 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 725 Links: 1 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-03-24 11:12:13.335698775 +0000 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-24 11:05:54.283574849 +0000 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-24 11:05:54.283574849 +0000 2024-03-24T11:12:26.691 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-24 11:05:54.283574849 +0000 2024-03-24T11:12:26.691 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-03-24T11:12:26.756 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-03-24T11:12:26.756 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-03-24T11:12:26.756 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000198172 s, 2.6 MB/s 2024-03-24T11:12:26.757 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-03-24T11:12:26.813 INFO:tasks.cephadm:Deploying osd.0 on smithi100 with /dev/vg_nvme/lv_4... 2024-03-24T11:12:26.813 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:12:27.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: Reconfiguring mgr.smithi100.smxgyi (unknown last config time)... 2024-03-24T11:12:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi100.smxgyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:12:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:12:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:27 smithi100 ceph-mon[28941]: Reconfiguring daemon mgr.smithi100.smxgyi on smithi100 2024-03-24T11:12:27.403 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: Reconfiguring mgr.smithi100.smxgyi (unknown last config time)... 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi100.smxgyi", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:27.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:27 smithi105 ceph-mon[36437]: Reconfiguring daemon mgr.smithi100.smxgyi on smithi100 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: Reconfiguring crash.smithi100 (monmap changed)... 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi100", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:28 smithi100 ceph-mon[28941]: Reconfiguring daemon crash.smithi100 on smithi100 2024-03-24T11:12:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: Reconfiguring crash.smithi100 (monmap changed)... 2024-03-24T11:12:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi100", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:12:28.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:28.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:28 smithi105 ceph-mon[36437]: Reconfiguring daemon crash.smithi100 on smithi100 2024-03-24T11:12:30.293 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:12:30.293 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:12:30.293 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-03-24T11:12:30.294 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-03-24T11:12:30.294 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0116472 s, 900 MB/s 2024-03-24T11:12:30.294 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-03-24T11:12:30.650 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:30 smithi100 ceph-mon[28941]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:30.650 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:30 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:30.650 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:30 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:30.650 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:30 smithi100 ceph-mon[28941]: Reconfiguring alertmanager.smithi100 (dependencies changed)... 2024-03-24T11:12:30.650 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:30 smithi100 ceph-mon[28941]: Reconfiguring daemon alertmanager.smithi100 on smithi100 2024-03-24T11:12:30.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:30 smithi105 ceph-mon[36437]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:30.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:30 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:30.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:30 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:30.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:30 smithi105 ceph-mon[36437]: Reconfiguring alertmanager.smithi100 (dependencies changed)... 2024-03-24T11:12:30.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:30 smithi105 ceph-mon[36437]: Reconfiguring daemon alertmanager.smithi100 on smithi100 2024-03-24T11:12:30.942 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi100:vg_nvme/lv_4 2024-03-24T11:12:31.796 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:12:32.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:32 smithi105 ceph-mon[36437]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:32.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:32 smithi100 ceph-mon[28941]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:34.565 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:34 smithi100 ceph-mon[28941]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:34.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 -- 172.21.15.100:0/757980731 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fcf70 msgr2=0x7f21340fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 --2- 172.21.15.100:0/757980731 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fcf70 0x7f21340fd3e0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7f2128004940 tx=0x7f2128009e30 comp rx=0 tx=0).stop 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 -- 172.21.15.100:0/757980731 shutdown_connections 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 --2- 172.21.15.100:0/757980731 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fcf70 0x7f21340fd3e0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 --2- 172.21.15.100:0/757980731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fbe00 0x7f21340fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 -- 172.21.15.100:0/757980731 >> 172.21.15.100:0/757980731 conn(0x7f21340f74e0 msgr2=0x7f21340f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 -- 172.21.15.100:0/757980731 shutdown_connections 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.590+0000 7f21393da700 1 -- 172.21.15.100:0/757980731 wait complete. 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f21393da700 1 Processor -- start 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f21393da700 1 -- start start 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f21393da700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 0x7f213410fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:34.593 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f21393da700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fcf70 0x7f2134110150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f21393da700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134110750 con 0x7f21340fbe00 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f21393da700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134110fa0 con 0x7f21340fcf70 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f2132ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 0x7f213410fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f2132ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 0x7f213410fc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47622/0 (socket says 172.21.15.100:47622) 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.591+0000 7f2132ffd700 1 -- 172.21.15.100:0/4147327532 learned_addr learned my addr 172.21.15.100:0/4147327532 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f21327fc700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fcf70 0x7f2134110150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f2132ffd700 1 -- 172.21.15.100:0/4147327532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fcf70 msgr2=0x7f2134110150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f2132ffd700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fcf70 0x7f2134110150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f2132ffd700 1 -- 172.21.15.100:0/4147327532 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2128005040 con 0x7f21340fbe00 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f21327fc700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fcf70 0x7f2134110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:12:34.594 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f2132ffd700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 0x7f213410fc10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2124002a50 tx=0x7f21240046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:34.595 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2124007cc0 con 0x7f21340fbe00 2024-03-24T11:12:34.595 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f212400f040 con 0x7f21340fbe00 2024-03-24T11:12:34.595 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2134111280 con 0x7f21340fbe00 2024-03-24T11:12:34.595 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2124009640 con 0x7f21340fbe00 2024-03-24T11:12:34.595 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.592+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21341117c0 con 0x7f21340fbe00 2024-03-24T11:12:34.596 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.593+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2134101900 con 0x7f21340fbe00 2024-03-24T11:12:34.596 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.593+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f212400d690 con 0x7f21340fbe00 2024-03-24T11:12:34.596 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.594+0000 7f211b7fe700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f211c069210 0x7f211c06b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:12:34.596 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.594+0000 7f21327fc700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f211c069210 0x7f211c06b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:12:34.596 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.594+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f2124085dc0 con 0x7f21340fbe00 2024-03-24T11:12:34.596 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.594+0000 7f21327fc700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f211c069210 0x7f211c06b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f212800b3b0 tx=0x7f2128009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:12:34.598 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.598+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2124056c60 con 0x7f21340fbe00 2024-03-24T11:12:34.796 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:12:34.795+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f2134111a30 con 0x7f211c069210 2024-03-24T11:12:34.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:34 smithi105 ceph-mon[36437]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:35.764 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:35 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:12:35.765 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:35 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:12:35.765 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:35 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:35.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:35 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:12:35.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:35 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:12:35.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:35 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:36.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:36 smithi100 ceph-mon[28941]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:12:36.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:36 smithi100 ceph-mon[28941]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:36.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:36 smithi105 ceph-mon[36437]: from='client.14244 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:12:36.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:36 smithi105 ceph-mon[36437]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:38.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:37 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:38.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:37 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:38.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:37 smithi105 ceph-mon[36437]: Reconfiguring grafana.smithi100 (dependencies changed)... 2024-03-24T11:12:38.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:37 smithi105 ceph-mon[36437]: Reconfiguring daemon grafana.smithi100 on smithi100 2024-03-24T11:12:38.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:37 smithi105 ceph-mon[36437]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:37 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:37 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:37 smithi100 ceph-mon[28941]: Reconfiguring grafana.smithi100 (dependencies changed)... 2024-03-24T11:12:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:37 smithi100 ceph-mon[28941]: Reconfiguring daemon grafana.smithi100 on smithi100 2024-03-24T11:12:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:37 smithi100 ceph-mon[28941]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:39.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:39 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3966445302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc6e0f08-af11-46b2-923f-17e549042b98"}]: dispatch 2024-03-24T11:12:39.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:39 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3966445302' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc6e0f08-af11-46b2-923f-17e549042b98"}]': finished 2024-03-24T11:12:39.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:39 smithi100 ceph-mon[28941]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:12:39.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:39 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:12:39.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:39 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:39.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:39 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3434945454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:12:39.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:39 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3966445302' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "cc6e0f08-af11-46b2-923f-17e549042b98"}]: dispatch 2024-03-24T11:12:39.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:39 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3966445302' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "cc6e0f08-af11-46b2-923f-17e549042b98"}]': finished 2024-03-24T11:12:39.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:39 smithi105 ceph-mon[36437]: osdmap e6: 1 total, 0 up, 1 in 2024-03-24T11:12:39.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:39 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:12:39.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:39 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:39.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:39 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3434945454' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:12:40.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:40 smithi105 ceph-mon[36437]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:40.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:40 smithi100 ceph-mon[28941]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:42.298 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:42 smithi100 ceph-mon[28941]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:42 smithi105 ceph-mon[36437]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:44 smithi100 ceph-mon[28941]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:44 smithi105 ceph-mon[36437]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:46.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:46 smithi100 ceph-mon[28941]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:46 smithi105 ceph-mon[36437]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:48.604 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:48 smithi100 ceph-mon[28941]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:48.604 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:48.604 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:48.604 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:48 smithi100 ceph-mon[28941]: Reconfiguring prometheus.smithi100 (dependencies changed)... 2024-03-24T11:12:48.604 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:48 smithi100 ceph-mon[28941]: Reconfiguring daemon prometheus.smithi100 on smithi100 2024-03-24T11:12:48.983 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:48 smithi105 ceph-mon[36437]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:48.983 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:48.983 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:48.983 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:48 smithi105 ceph-mon[36437]: Reconfiguring prometheus.smithi100 (dependencies changed)... 2024-03-24T11:12:48.983 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:48 smithi105 ceph-mon[36437]: Reconfiguring daemon prometheus.smithi100 on smithi100 2024-03-24T11:12:49.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:12:49.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-03-24T11:12:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:50.746 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:50 smithi100 ceph-mon[28941]: Deploying daemon osd.0 on smithi100 2024-03-24T11:12:50.746 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:50 smithi100 ceph-mon[28941]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:51.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:50 smithi105 ceph-mon[36437]: Deploying daemon osd.0 on smithi100 2024-03-24T11:12:51.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:50 smithi105 ceph-mon[36437]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:52.891 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:52 smithi100 ceph-mon[28941]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:53.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:52 smithi105 ceph-mon[36437]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:54.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:54.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:54.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:54.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: Reconfiguring crash.smithi105 (monmap changed)... 2024-03-24T11:12:54.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:12:54.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:54.297 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: Reconfiguring daemon crash.smithi105 on smithi105 2024-03-24T11:12:54.297 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:54.447 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:54.447 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:54.447 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:54.447 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: Reconfiguring crash.smithi105 (monmap changed)... 2024-03-24T11:12:54.447 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-03-24T11:12:54.447 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:54.448 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: Reconfiguring daemon crash.smithi105 on smithi105 2024-03-24T11:12:54.448 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:54 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:12:55.595 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:55.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:55.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: Reconfiguring mgr.smithi105.nmspmy (monmap changed)... 2024-03-24T11:12:55.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:12:55.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:12:55.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:55.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:55 smithi100 ceph-mon[28941]: Reconfiguring daemon mgr.smithi105.nmspmy on smithi105 2024-03-24T11:12:55.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:55.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:55.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: Reconfiguring mgr.smithi105.nmspmy (monmap changed)... 2024-03-24T11:12:55.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:12:55.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:12:55.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:55.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:55 smithi105 ceph-mon[36437]: Reconfiguring daemon mgr.smithi105.nmspmy on smithi105 2024-03-24T11:12:56.784 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:56 smithi105 ceph-mon[36437]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:56.784 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:56.784 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:56.784 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:12:56.784 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:12:56.785 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:56 smithi100 ceph-mon[28941]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-03-24T11:12:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-03-24T11:12:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:12:57.768 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: Reconfiguring mon.smithi105 (monmap changed)... 2024-03-24T11:12:57.768 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: Reconfiguring daemon mon.smithi105 on smithi105 2024-03-24T11:12:57.768 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.768 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.768 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:12:57.768 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi100.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi100.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:57 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: Reconfiguring mon.smithi105 (monmap changed)... 2024-03-24T11:12:57.769 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: Reconfiguring daemon mon.smithi105 on smithi105 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi100.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi100.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:57.770 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:57 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi100.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi100.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:58.558 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:12:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:58.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:12:58.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-03-24T11:12:58.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi100.front.sepia.ceph.com:9093"}]: dispatch 2024-03-24T11:12:58.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-03-24T11:12:58.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi100.front.sepia.ceph.com:3000"}]: dispatch 2024-03-24T11:12:58.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-03-24T11:12:58.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi100.front.sepia.ceph.com:9095"}]: dispatch 2024-03-24T11:12:58.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:12:58.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:12:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:00.306 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:00 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:00.306 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:00 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:00.306 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:00 smithi105 ceph-mon[36437]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:00.360 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:00.360 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:00.360 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:00 smithi100 ceph-mon[28941]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:02.429 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:02 smithi100 ceph-mon[28941]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:02.430 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:02 smithi100 ceph-mon[28941]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:13:02.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.490+0000 7f211b7fe700 1 -- 172.21.15.100:0/4147327532 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f2134111a30 con 0x7f211c069210 2024-03-24T11:13:02.491 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 0 on host 'smithi100' 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f211c069210 msgr2=0x7f211c06b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f211c069210 0x7f211c06b6c0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f212800b3b0 tx=0x7f2128009e30 comp rx=0 tx=0).stop 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 msgr2=0x7f213410fc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 0x7f213410fc10 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7f2124002a50 tx=0x7f21240046b0 comp rx=0 tx=0).stop 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 shutdown_connections 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f211c069210 0x7f211c06b6c0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f21340fbe00 0x7f213410fc10 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 --2- 172.21.15.100:0/4147327532 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f21340fcf70 0x7f2134110150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 >> 172.21.15.100:0/4147327532 conn(0x7f21340f74e0 msgr2=0x7f21340f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 shutdown_connections 2024-03-24T11:13:02.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:02.492+0000 7f21393da700 1 -- 172.21.15.100:0/4147327532 wait complete. 2024-03-24T11:13:02.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:02 smithi105 ceph-mon[36437]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:02.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:02 smithi105 ceph-mon[36437]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-03-24T11:13:03.215 DEBUG:teuthology.orchestra.run.smithi100:osd.0> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.0.service 2024-03-24T11:13:03.217 INFO:tasks.cephadm:Deploying osd.1 on smithi100 with /dev/vg_nvme/lv_3... 2024-03-24T11:13:03.218 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:03.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:03.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-03-24T11:13:03.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: osdmap e7: 1 total, 0 up, 1 in 2024-03-24T11:13:03.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:03.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-03-24T11:13:03.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:03.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:03.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-03-24T11:13:03.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: osdmap e8: 1 total, 0 up, 1 in 2024-03-24T11:13:03.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:03.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:03.746 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:13:04.745 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:04 smithi100 ceph-mon[28941]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:04.745 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:04.746 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:04 smithi100 ceph-mon[28941]: osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] boot 2024-03-24T11:13:04.746 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:04 smithi100 ceph-mon[28941]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:13:04.746 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:04.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:04 smithi105 ceph-mon[36437]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:04.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:04.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:04 smithi105 ceph-mon[36437]: osd.0 [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] boot 2024-03-24T11:13:04.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:04 smithi105 ceph-mon[36437]: osdmap e9: 1 total, 1 up, 1 in 2024-03-24T11:13:04.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-03-24T11:13:05.761 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:05 smithi100 ceph-mon[28941]: purged_snaps scrub starts 2024-03-24T11:13:05.761 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:05 smithi100 ceph-mon[28941]: purged_snaps scrub ok 2024-03-24T11:13:05.761 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:05 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:05.761 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:05 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:05 smithi105 ceph-mon[36437]: purged_snaps scrub starts 2024-03-24T11:13:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:05 smithi105 ceph-mon[36437]: purged_snaps scrub ok 2024-03-24T11:13:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:05 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:05 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:06.336 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:13:06.337 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:13:06.337 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-03-24T11:13:06.337 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-03-24T11:13:06.337 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124057 s, 845 MB/s 2024-03-24T11:13:06.337 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-03-24T11:13:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:06 smithi105 ceph-mon[36437]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:06 smithi105 ceph-mon[36437]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T11:13:06.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:06 smithi100 ceph-mon[28941]: pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-03-24T11:13:06.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:06 smithi100 ceph-mon[28941]: osdmap e10: 1 total, 1 up, 1 in 2024-03-24T11:13:06.985 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi100:vg_nvme/lv_3 2024-03-24T11:13:08.326 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:13:08.725 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:08 smithi100 ceph-mon[28941]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:08.725 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:08.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:08 smithi105 ceph-mon[36437]: pgmap v35: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:08.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2308036644 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 msgr2=0x7f3c5c0fb610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2308036644 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fb610 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c004940 tx=0x7f3c4c009e30 comp rx=0 tx=0).stop 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2308036644 shutdown_connections 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2308036644 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 0x7f3c5c102380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2308036644 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fb610 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2308036644 >> 172.21.15.100:0/2308036644 conn(0x7f3c5c0f6ba0 msgr2=0x7f3c5c0f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2308036644 shutdown_connections 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2308036644 wait complete. 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 Processor -- start 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- start start 2024-03-24T11:13:10.555 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fdd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 0x7f3c5c0fc1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0fc790 con 0x7f3c5c0fb200 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c60ca8700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c5c0fc900 con 0x7f3c5c0fbb50 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c5a59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fdd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.553+0000 7f3c5a59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fdd10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:46568/0 (socket says 172.21.15.100:46568) 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c5a59c700 1 -- 172.21.15.100:0/2158525835 learned_addr learned my addr 172.21.15.100:0/2158525835 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c59d9b700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 0x7f3c5c0fc1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c5a59c700 1 -- 172.21.15.100:0/2158525835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 msgr2=0x7f3c5c0fc1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c5a59c700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 0x7f3c5c0fc1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:10.556 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c5a59c700 1 -- 172.21.15.100:0/2158525835 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c4c005040 con 0x7f3c5c0fb200 2024-03-24T11:13:10.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c59d9b700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 0x7f3c5c0fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T11:13:10.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c5a59c700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fdd10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00b3b0 tx=0x7f3c4c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:13:10.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4c00e040 con 0x7f3c5c0fb200 2024-03-24T11:13:10.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3c4c00bbf0 con 0x7f3c5c0fb200 2024-03-24T11:13:10.557 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.554+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c5c0fcb80 con 0x7f3c5c0fb200 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.555+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3c4c01c630 con 0x7f3c5c0fb200 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.555+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c5c115a50 con 0x7f3c5c0fb200 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.556+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7f3c4c00bd60 con 0x7f3c5c0fb200 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.556+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c5c05f5f0 con 0x7f3c5c0fb200 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.556+0000 7f3c4affd700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3c44069210 0x7f3c4406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.556+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7f3c4c086f90 con 0x7f3c5c0fb200 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.556+0000 7f3c59d9b700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3c44069210 0x7f3c4406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:13:10.558 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.557+0000 7f3c59d9b700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3c44069210 0x7f3c4406b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3c50004fd0 tx=0x7f3c5000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:13:10.565 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.565+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3c4c0b9190 con 0x7f3c5c0fb200 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: Detected new or changed devices on smithi100 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: Adjusting osd_memory_target on smithi100 to 12864M 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:10.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:10.767+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f3c5c101be0 con 0x7f3c44069210 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: pgmap v36: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: Detected new or changed devices on smithi100 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: Adjusting osd_memory_target on smithi100 to 12864M 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:10.849 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:10.850 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:13:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:13:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:11.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:11 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:12.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:11 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:13:12.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:11 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:13:12.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:11 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:12.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:11 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:12.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:11 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:12.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:12 smithi100 ceph-mon[28941]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:12.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:12 smithi100 ceph-mon[28941]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:13.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:12 smithi105 ceph-mon[36437]: from='client.14262 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:13.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:12 smithi105 ceph-mon[36437]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:14.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:14 smithi100 ceph-mon[28941]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:15.016 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:14 smithi105 ceph-mon[36437]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:15.626 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:15 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3843085703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb"}]: dispatch 2024-03-24T11:13:15.626 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:15 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3843085703' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb"}]': finished 2024-03-24T11:13:15.626 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:15 smithi100 ceph-mon[28941]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:13:15.626 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:15 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:16.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3843085703' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb"}]: dispatch 2024-03-24T11:13:16.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:15 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3843085703' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb"}]': finished 2024-03-24T11:13:16.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:15 smithi105 ceph-mon[36437]: osdmap e11: 2 total, 1 up, 2 in 2024-03-24T11:13:16.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3458641570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:17.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:17.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3458641570' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:13:17.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:17.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:17.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:17.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:17.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:18.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:18 smithi100 ceph-mon[28941]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:19.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:18 smithi105 ceph-mon[36437]: pgmap v41: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:20.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:20 smithi100 ceph-mon[28941]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:20.913 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:20 smithi105 ceph-mon[36437]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:22.879 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:22 smithi100 ceph-mon[28941]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:22.879 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:22 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:13:22.879 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:22 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:23.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:22 smithi105 ceph-mon[36437]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:23.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:22 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-03-24T11:13:23.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:22 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:23.809 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:23 smithi100 ceph-mon[28941]: Deploying daemon osd.1 on smithi100 2024-03-24T11:13:23.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:24.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:23 smithi105 ceph-mon[36437]: Deploying daemon osd.1 on smithi100 2024-03-24T11:13:24.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:24.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:24 smithi100 ceph-mon[28941]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:25.016 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:24 smithi105 ceph-mon[36437]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:26.873 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:26 smithi100 ceph-mon[28941]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:27.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:26 smithi105 ceph-mon[36437]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:28.678 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:28 smithi100 ceph-mon[28941]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:28.679 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:28 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:28 smithi105 ceph-mon[36437]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:28 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:30.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:30 smithi100 ceph-mon[28941]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:30.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:30 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:30.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:30 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:31.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:30 smithi105 ceph-mon[36437]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:31.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:30 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:31.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:30 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:31.301 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 1 on host 'smithi100' 2024-03-24T11:13:31.301 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.300+0000 7f3c4affd700 1 -- 172.21.15.100:0/2158525835 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f3c5c101be0 con 0x7f3c44069210 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.301+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3c44069210 msgr2=0x7f3c4406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.301+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3c44069210 0x7f3c4406b6c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f3c50004fd0 tx=0x7f3c5000b040 comp rx=0 tx=0).stop 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.301+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 msgr2=0x7f3c5c0fdd10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.301+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fdd10 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 crypto rx=0x7f3c4c00b3b0 tx=0x7f3c4c009e30 comp rx=0 tx=0).stop 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 shutdown_connections 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3c44069210 0x7f3c4406b6c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3c5c0fb200 0x7f3c5c0fdd10 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 --2- 172.21.15.100:0/2158525835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3c5c0fbb50 0x7f3c5c0fc1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:31.303 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 >> 172.21.15.100:0/2158525835 conn(0x7f3c5c0f6ba0 msgr2=0x7f3c5c1004c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:13:31.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 shutdown_connections 2024-03-24T11:13:31.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:31.302+0000 7f3c60ca8700 1 -- 172.21.15.100:0/2158525835 wait complete. 2024-03-24T11:13:31.923 DEBUG:teuthology.orchestra.run.smithi100:osd.1> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.1.service 2024-03-24T11:13:31.925 INFO:tasks.cephadm:Deploying osd.2 on smithi100 with /dev/vg_nvme/lv_2... 2024-03-24T11:13:31.925 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:13:32.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:32 smithi105 ceph-mon[36437]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:32.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:32 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:32.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:32 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:32.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:32 smithi105 ceph-mon[36437]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:13:32.560 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:32 smithi100 ceph-mon[28941]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:32.560 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:32 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:32.560 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:32 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:32.560 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:32 smithi100 ceph-mon[28941]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-03-24T11:13:32.966 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:13:33.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:33 smithi100 ceph-mon[28941]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:13:33.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:33 smithi100 ceph-mon[28941]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:13:33.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:33 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:33.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:33 smithi100 ceph-mon[28941]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-03-24T11:13:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:13:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:13:33.643+0000 7fc6db43b700 -1 osd.1 0 waiting for initial osdmap 2024-03-24T11:13:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:13:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:13:33.647+0000 7fc6d4a33700 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:13:34.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:33 smithi105 ceph-mon[36437]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-03-24T11:13:34.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:33 smithi105 ceph-mon[36437]: osdmap e12: 2 total, 1 up, 2 in 2024-03-24T11:13:34.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:33 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:34.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:33 smithi105 ceph-mon[36437]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-03-24T11:13:34.910 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:34 smithi100 ceph-mon[28941]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:34.911 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:34 smithi100 ceph-mon[28941]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-03-24T11:13:34.911 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:34 smithi100 ceph-mon[28941]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T11:13:34.911 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:34 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:34.911 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:34 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:35.016 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:34 smithi105 ceph-mon[36437]: pgmap v50: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:35.016 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:34 smithi105 ceph-mon[36437]: from='osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-03-24T11:13:35.016 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:34 smithi105 ceph-mon[36437]: osdmap e13: 2 total, 1 up, 2 in 2024-03-24T11:13:35.016 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:34 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:35.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:34 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:36.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:35 smithi105 ceph-mon[36437]: purged_snaps scrub starts 2024-03-24T11:13:36.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:35 smithi105 ceph-mon[36437]: purged_snaps scrub ok 2024-03-24T11:13:36.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:35 smithi105 ceph-mon[36437]: osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] boot 2024-03-24T11:13:36.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:35 smithi105 ceph-mon[36437]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:13:36.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:35 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:36.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:35 smithi100 ceph-mon[28941]: purged_snaps scrub starts 2024-03-24T11:13:36.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:35 smithi100 ceph-mon[28941]: purged_snaps scrub ok 2024-03-24T11:13:36.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:35 smithi100 ceph-mon[28941]: osd.1 [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] boot 2024-03-24T11:13:36.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:35 smithi100 ceph-mon[28941]: osdmap e14: 2 total, 2 up, 2 in 2024-03-24T11:13:36.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:35 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-03-24T11:13:36.192 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:13:36.192 INFO:teuthology.orchestra.run.smithi100.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:13:36.192 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10+0 records in 2024-03-24T11:13:36.192 INFO:teuthology.orchestra.run.smithi100.stderr:10+0 records out 2024-03-24T11:13:36.192 INFO:teuthology.orchestra.run.smithi100.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0135733 s, 773 MB/s 2024-03-24T11:13:36.192 INFO:teuthology.orchestra.run.smithi100.stderr:--> Zapping successful for: 2024-03-24T11:13:36.874 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:36 smithi100 ceph-mon[28941]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:36.874 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:36 smithi100 ceph-mon[28941]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T11:13:36.898 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi100:vg_nvme/lv_2 2024-03-24T11:13:37.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:36 smithi105 ceph-mon[36437]: pgmap v53: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-03-24T11:13:37.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:36 smithi105 ceph-mon[36437]: osdmap e15: 2 total, 2 up, 2 in 2024-03-24T11:13:37.342 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:13:38.509 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:38 smithi100 ceph-mon[28941]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:38.509 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:38.509 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:38.510 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:38.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:38 smithi105 ceph-mon[36437]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:38.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:38.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:38.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:40.488 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 -- 172.21.15.100:0/1706780232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0662e0 msgr2=0x7ff05c0ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 --2- 172.21.15.100:0/1706780232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0662e0 0x7ff05c0ff9c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 crypto rx=0x7ff050004940 tx=0x7ff050009e30 comp rx=0 tx=0).stop 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 -- 172.21.15.100:0/1706780232 shutdown_connections 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 --2- 172.21.15.100:0/1706780232 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff05c0fff00 0x7ff05c102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 --2- 172.21.15.100:0/1706780232 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0662e0 0x7ff05c0ff9c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 -- 172.21.15.100:0/1706780232 >> 172.21.15.100:0/1706780232 conn(0x7ff05c0f7430 msgr2=0x7ff05c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 -- 172.21.15.100:0/1706780232 shutdown_connections 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 -- 172.21.15.100:0/1706780232 wait complete. 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.471+0000 7ff0626e5700 1 Processor -- start 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff0626e5700 1 -- start start 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff0626e5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff05c0662e0 0x7ff05c10da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff0626e5700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 0x7ff05c10df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff0626e5700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c10e560 con 0x7ff05c0fff00 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff0626e5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff05c10ed60 con 0x7ff05c0662e0 2024-03-24T11:13:40.489 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 0x7ff05c10df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 0x7ff05c10df60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53096/0 (socket says 172.21.15.100:53096) 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05bfff700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff05c0662e0 0x7ff05c10da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 -- 172.21.15.100:0/173586916 learned_addr learned my addr 172.21.15.100:0/173586916 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 -- 172.21.15.100:0/173586916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff05c0662e0 msgr2=0x7ff05c10da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff05c0662e0 0x7ff05c10da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 -- 172.21.15.100:0/173586916 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff050005040 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.472+0000 7ff05b7fe700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 0x7ff05c10df60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff04c002a80 tx=0x7ff04c0046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.473+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04c007cc0 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.473+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff04c00f040 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.473+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff05c10f040 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.473+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff04c00dac0 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.473+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff05c10f580 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.474+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff05c05f5f0 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.474+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7ff04c00dc20 con 0x7ff05c0fff00 2024-03-24T11:13:40.490 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.474+0000 7ff058ff9700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff044069210 0x7ff04406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:13:40.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.475+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7ff04c086140 con 0x7ff05c0fff00 2024-03-24T11:13:40.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.475+0000 7ff05bfff700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff044069210 0x7ff04406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:13:40.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.475+0000 7ff05bfff700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff044069210 0x7ff04406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff05000b970 tx=0x7ff050009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:13:40.491 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.479+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff04c056bd0 con 0x7ff05c0fff00 2024-03-24T11:13:40.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:40 smithi100 ceph-mon[28941]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:40.682 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:13:40.682+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7ff05c10f8b0 con 0x7ff044069210 2024-03-24T11:13:40.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:40 smithi105 ceph-mon[36437]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:41.669 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:41 smithi100 ceph-mon[28941]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:41.669 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:41 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:13:41.669 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:41 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:13:41.669 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:41 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:41.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:41 smithi105 ceph-mon[36437]: from='client.14280 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi100:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:13:41.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:41 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:13:41.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:41 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:13:41.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:41 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:42.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: Detected new or changed devices on smithi100 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: Adjusting osd_memory_target on smithi100 to 6432M 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: Detected new or changed devices on smithi100 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: Adjusting osd_memory_target on smithi100 to 6432M 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.810 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:42.811 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:42.811 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:42.811 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:44.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:44 smithi100 ceph-mon[28941]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:44.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:44 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1013854841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44db3f0c-2877-4026-93f3-febdb7f46d67"}]: dispatch 2024-03-24T11:13:44.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:44 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1013854841' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44db3f0c-2877-4026-93f3-febdb7f46d67"}]': finished 2024-03-24T11:13:44.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:44 smithi100 ceph-mon[28941]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:13:44.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:13:44.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:44 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2928741165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:13:45.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:44 smithi105 ceph-mon[36437]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:45.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:44 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/1013854841' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "44db3f0c-2877-4026-93f3-febdb7f46d67"}]: dispatch 2024-03-24T11:13:45.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:44 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/1013854841' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "44db3f0c-2877-4026-93f3-febdb7f46d67"}]': finished 2024-03-24T11:13:45.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:44 smithi105 ceph-mon[36437]: osdmap e16: 3 total, 2 up, 3 in 2024-03-24T11:13:45.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:13:45.017 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:44 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2928741165' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:13:46.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:46 smithi100 ceph-mon[28941]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:47.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:46 smithi105 ceph-mon[36437]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:48.502 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:48 smithi100 ceph-mon[28941]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:48.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:48.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:48.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:48.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:48.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:48 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:48 smithi105 ceph-mon[36437]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:13:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:48 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:50.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:50 smithi100 ceph-mon[28941]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:50.505 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:50 smithi105 ceph-mon[36437]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:52.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:52 smithi100 ceph-mon[28941]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:52.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:52 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:13:52.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:52 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:52.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:52 smithi100 ceph-mon[28941]: Deploying daemon osd.2 on smithi100 2024-03-24T11:13:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:52 smithi105 ceph-mon[36437]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:52 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-03-24T11:13:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:52 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:13:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:52 smithi105 ceph-mon[36437]: Deploying daemon osd.2 on smithi100 2024-03-24T11:13:54.314 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:54 smithi100 ceph-mon[28941]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:54.314 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:54 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:54 smithi105 ceph-mon[36437]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:54 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:13:56.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:56 smithi100 ceph-mon[28941]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:56.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:56 smithi105 ceph-mon[36437]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:58.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:58.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:58.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:58.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:13:58 smithi105 ceph-mon[36437]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:13:58.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:13:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:13:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:13:58 smithi100 ceph-mon[28941]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:00.331 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:00 smithi100 ceph-mon[28941]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:00.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:00 smithi105 ceph-mon[36437]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:00.689 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.689+0000 7ff058ff9700 1 -- 172.21.15.100:0/173586916 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7ff05c10f8b0 con 0x7ff044069210 2024-03-24T11:14:00.690 INFO:teuthology.orchestra.run.smithi100.stdout:Created osd(s) 2 on host 'smithi100' 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff044069210 msgr2=0x7ff04406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff044069210 0x7ff04406b6c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7ff05000b970 tx=0x7ff050009e30 comp rx=0 tx=0).stop 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 msgr2=0x7ff05c10df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 0x7ff05c10df60 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7ff04c002a80 tx=0x7ff04c0046b0 comp rx=0 tx=0).stop 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 shutdown_connections 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff044069210 0x7ff04406b6c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff05c0662e0 0x7ff05c10da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 --2- 172.21.15.100:0/173586916 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff05c0fff00 0x7ff05c10df60 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 >> 172.21.15.100:0/173586916 conn(0x7ff05c0f7430 msgr2=0x7ff05c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 shutdown_connections 2024-03-24T11:14:00.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:14:00.691+0000 7ff0626e5700 1 -- 172.21.15.100:0/173586916 wait complete. 2024-03-24T11:14:01.358 DEBUG:teuthology.orchestra.run.smithi100:osd.2> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.2.service 2024-03-24T11:14:01.360 INFO:tasks.cephadm:Deploying osd.3 on smithi105 with /dev/vg_nvme/lv_4... 2024-03-24T11:14:01.360 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-03-24T11:14:01.619 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:14:01.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:01 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:01.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:01 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:01.841 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:01 smithi100 ceph-mon[28941]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:14:01.901 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:01 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:01.902 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:01 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:01.902 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:01 smithi105 ceph-mon[36437]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-03-24T11:14:02.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:02 smithi105 ceph-mon[36437]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:02.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:02 smithi105 ceph-mon[36437]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:14:02.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:02 smithi105 ceph-mon[36437]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:14:02.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:02 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:02.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:02 smithi105 ceph-mon[36437]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-03-24T11:14:03.106 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-mon[28941]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:03.106 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-mon[28941]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-03-24T11:14:03.106 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-mon[28941]: osdmap e17: 3 total, 2 up, 3 in 2024-03-24T11:14:03.106 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:03.106 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-mon[28941]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]: dispatch 2024-03-24T11:14:03.107 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:14:02.660+0000 7fd4f6a0b700 -1 osd.2 0 waiting for initial osdmap 2024-03-24T11:14:03.107 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:14:02.664+0000 7fd4f1806700 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:14:03.397 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-03-24T11:14:03.397 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-03-24T11:14:03.397 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-03-24T11:14:03.397 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-03-24T11:14:03.397 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011481 s, 913 MB/s 2024-03-24T11:14:03.397 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-03-24T11:14:03.915 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:03 smithi100 ceph-mon[28941]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-03-24T11:14:03.915 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:03 smithi100 ceph-mon[28941]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T11:14:03.915 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:03.915 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:03 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:03.961 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:03 smithi105 ceph-mon[36437]: from='osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi100", "root=default"]}]': finished 2024-03-24T11:14:03.961 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:03 smithi105 ceph-mon[36437]: osdmap e18: 3 total, 2 up, 3 in 2024-03-24T11:14:03.961 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:03.962 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:03 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:03.986 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-03-24T11:14:04.245 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:14:04.918 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: purged_snaps scrub starts 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: purged_snaps scrub ok 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] boot 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:04.919 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: purged_snaps scrub starts 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: purged_snaps scrub ok 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: osd.2 [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] boot 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: osdmap e19: 3 total, 3 up, 3 in 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:05.010 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:05.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.721+0000 7fa3e0abc700 1 -- 172.21.15.105:0/2614258866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 msgr2=0x7fa3dc0fe4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:05.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/2614258866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc0fe4b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7fa3c4004940 tx=0x7fa3c4009e30 comp rx=0 tx=0).stop 2024-03-24T11:14:05.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- 172.21.15.105:0/2614258866 shutdown_connections 2024-03-24T11:14:05.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/2614258866 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc0fe4b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:05.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/2614258866 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3dc0fce40 0x7fa3dc0fd250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:05.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- 172.21.15.105:0/2614258866 >> 172.21.15.105:0/2614258866 conn(0x7fa3dc0f8550 msgr2=0x7fa3dc0fa980 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- 172.21.15.105:0/2614258866 shutdown_connections 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- 172.21.15.105:0/2614258866 wait complete. 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 Processor -- start 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- start start 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3dc0fce40 0x7fa3dc10d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc10de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc10e470 con 0x7fa3dc0fce40 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.722+0000 7fa3e0abc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3dc10e5e0 con 0x7fa3dc0fe040 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc10de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc10de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47540/0 (socket says 172.21.15.105:47540) 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 -- 172.21.15.105:0/1027219239 learned_addr learned my addr 172.21.15.105:0/1027219239 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3da59c700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3dc0fce40 0x7fa3dc10d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 -- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3dc0fce40 msgr2=0x7fa3dc10d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:05.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3dc0fce40 0x7fa3dc10d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 -- 172.21.15.105:0/1027219239 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c4005040 con 0x7fa3dc0fe040 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d1d9b700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc10de70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400b3b0 tx=0x7fa3c4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c400e040 con 0x7fa3dc0fe040 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fa3c400bbf0 con 0x7fa3dc0fe040 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3dc112fe0 con 0x7fa3dc0fe040 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa3c401c630 con 0x7fa3dc0fe040 2024-03-24T11:14:05.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.723+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3dc113570 con 0x7fa3dc0fe040 2024-03-24T11:14:05.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.724+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3dc04c710 con 0x7fa3dc0fe040 2024-03-24T11:14:05.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.725+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 86567+0+0 (secure 0 0 0) 0x7fa3c400bd60 con 0x7fa3dc0fe040 2024-03-24T11:14:05.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.725+0000 7fa3d37fe700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fa3c8069210 0x7fa3c806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:05.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.725+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7fa3c4087540 con 0x7fa3dc0fe040 2024-03-24T11:14:05.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.725+0000 7fa3da59c700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fa3c8069210 0x7fa3c806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:05.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.726+0000 7fa3da59c700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fa3c8069210 0x7fa3c806b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0fdbb0 tx=0x7fa3cc00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:14:05.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.730+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fa3c4057c00 con 0x7fa3dc0fe040 2024-03-24T11:14:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:05 smithi105 ceph-mon[36437]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:14:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:05 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:14:05.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:05.923+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3dc113760 con 0x7fa3c8069210 2024-03-24T11:14:06.013 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:05 smithi100 ceph-mon[28941]: osdmap e20: 3 total, 3 up, 3 in 2024-03-24T11:14:06.014 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:05 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-03-24T11:14:06.944 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: pgmap v74: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:06.944 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:14:06.944 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:14:06.944 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:14:06.944 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:14:06.944 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:14:06.945 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:06.974 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: pgmap v74: 0 pgs: ; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:06.974 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-03-24T11:14:06.974 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: osdmap e21: 3 total, 3 up, 3 in 2024-03-24T11:14:06.974 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-03-24T11:14:06.974 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:14:06.974 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:14:06.975 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:07.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67777]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:14:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67777]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67777]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67777]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:07 smithi105 ceph-mon[36437]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:07 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:14:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:07 smithi105 ceph-mon[36437]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:14:08.011 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:07 smithi100 ceph-mon[28941]: from='client.24129 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:08.011 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:07 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-03-24T11:14:08.011 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:07 smithi100 ceph-mon[28941]: osdmap e22: 3 total, 3 up, 3 in 2024-03-24T11:14:08.011 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67780]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:14:08.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67780]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:08.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67780]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:08.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:14:07 smithi100 sudo[67780]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:08.291 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67881]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:14:08.291 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67881]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:08.291 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67881]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:08.291 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67881]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:08.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67969]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:14:08.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67969]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:08.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67969]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:08.545 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67969]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:08.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:08.762+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fa3c4013980 con 0x7fa3dc0fe040 2024-03-24T11:14:08.806 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67991]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:14:08.807 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67991]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:08.807 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67991]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:08.807 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67991]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:08.807 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:08.807 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:14:08.807 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: Detected new or changed devices on smithi100 2024-03-24T11:14:08.807 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:08.807 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:08.807 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: Adjusting osd_memory_target on smithi100 to 4288M 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/2115685239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f6ededc-bddf-4308-a416-80ae19ad3fa4"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f6ededc-bddf-4308-a416-80ae19ad3fa4"}]: dispatch 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f6ededc-bddf-4308-a416-80ae19ad3fa4"}]': finished 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:14:08.808 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:09.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: pgmap v77: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:09.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: osdmap e23: 3 total, 3 up, 3 in 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: Detected new or changed devices on smithi100 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: Adjusting osd_memory_target on smithi100 to 4288M 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:09.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/2115685239' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f6ededc-bddf-4308-a416-80ae19ad3fa4"}]: dispatch 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "1f6ededc-bddf-4308-a416-80ae19ad3fa4"}]: dispatch 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "1f6ededc-bddf-4308-a416-80ae19ad3fa4"}]': finished 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: osdmap e24: 4 total, 3 up, 4 in 2024-03-24T11:14:09.054 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:09.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67994]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:14:09.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67994]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:09.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67994]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:09.064 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:14:08 smithi100 sudo[67994]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:09.325 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68079]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:14:09.325 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68079]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:09.325 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68079]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:09.325 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68079]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:09.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68103]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:14:09.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68103]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:09.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68103]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:09.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 sudo[68103]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:09.840 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40002]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-03-24T11:14:09.840 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40002]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:09.841 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40002]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:09.841 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40002]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:10.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: mgrmap e19: smithi100.smxgyi(active, since 2m), standbys: smithi105.nmspmy 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/434945901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:14:10.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:09 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:14:10.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40193]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40193]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40193]: pam_unix(sudo:session): session opened for user root by (uid=167) 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 sudo[40193]: pam_unix(sudo:session): session closed for user root 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: mgrmap e19: smithi100.smxgyi(active, since 2m), standbys: smithi105.nmspmy 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/434945901' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: pgmap v80: 1 pgs: 1 unknown; 0 B data, 871 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:14:10.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:14:10.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi100"}]: dispatch 2024-03-24T11:14:10.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:09 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-03-24T11:14:11.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:10 smithi100 ceph-mon[28941]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:14:11.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:10 smithi100 ceph-mon[28941]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:14:11.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:10 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:11.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:10 smithi105 ceph-mon[36437]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-03-24T11:14:11.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:10 smithi105 ceph-mon[36437]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-03-24T11:14:11.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:10 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:12.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:11 smithi100 ceph-mon[28941]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:12.182 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:11 smithi105 ceph-mon[36437]: pgmap v81: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:13.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:13 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:13.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:13 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:13.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:13 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:13.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:13 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:13.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:13 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:13.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:13 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:13 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:13 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:13 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:13 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:14.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:14 smithi105 ceph-mon[36437]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:14.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:14 smithi100 ceph-mon[28941]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:15.780 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:14:15.780 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:15.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:15 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-03-24T11:14:15.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:15 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:16.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:16 smithi105 ceph-mon[36437]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:16.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:16 smithi105 ceph-mon[36437]: Deploying daemon osd.3 on smithi105 2024-03-24T11:14:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:16 smithi100 ceph-mon[28941]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:16.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:16 smithi100 ceph-mon[28941]: Deploying daemon osd.3 on smithi105 2024-03-24T11:14:18.799 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:18 smithi105 ceph-mon[36437]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:18.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:18 smithi100 ceph-mon[28941]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:20.773 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:20 smithi105 ceph-mon[36437]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:20.773 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:20.773 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:20.773 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:20.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:20 smithi100 ceph-mon[28941]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:20.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:20.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:20.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:22.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:22 smithi100 ceph-mon[28941]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:22.928 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:22 smithi105 ceph-mon[36437]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:23.650 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:23.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:23.946 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.947+0000 7fa3d37fe700 1 -- 172.21.15.105:0/1027219239 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fa3dc113760 con 0x7fa3c8069210 2024-03-24T11:14:23.947 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 3 on host 'smithi105' 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fa3c8069210 msgr2=0x7fa3c806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fa3c8069210 0x7fa3c806b6c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fa3dc0fdbb0 tx=0x7fa3cc00b040 comp rx=0 tx=0).stop 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 msgr2=0x7fa3dc10de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc10de70 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fa3c400b3b0 tx=0x7fa3c4009e30 comp rx=0 tx=0).stop 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 shutdown_connections 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fa3c8069210 0x7fa3c806b6c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fa3dc0fce40 0x7fa3dc10d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 --2- 172.21.15.105:0/1027219239 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa3dc0fe040 0x7fa3dc10de70 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 >> 172.21.15.105:0/1027219239 conn(0x7fa3dc0f8550 msgr2=0x7fa3dc0f9c60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 shutdown_connections 2024-03-24T11:14:23.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:23.949+0000 7fa3e0abc700 1 -- 172.21.15.105:0/1027219239 wait complete. 2024-03-24T11:14:24.517 DEBUG:teuthology.orchestra.run.smithi105:osd.3> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.3.service 2024-03-24T11:14:24.519 INFO:tasks.cephadm:Deploying osd.4 on smithi105 with /dev/vg_nvme/lv_3... 2024-03-24T11:14:24.519 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-03-24T11:14:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: from='osd.3 [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:14:24.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:14:24.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:24 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.831 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:14:24.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:24.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: from='osd.3 [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:14:24.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-03-24T11:14:24.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:24.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:24 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:26.241 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-mon[36437]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:14:26.241 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-mon[36437]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:14:26.241 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:26.241 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-mon[36437]: from='osd.3 [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:26.241 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-mon[36437]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:26.241 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-mon[36437]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:26.241 INFO:journalctl@ceph.osd.3.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-3[42006]: 2024-03-24T11:14:25.936+0000 7fe0098ee700 -1 osd.3 0 waiting for initial osdmap 2024-03-24T11:14:26.241 INFO:journalctl@ceph.osd.3.smithi105.stdout:Mar 24 11:14:25 smithi105 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-3[42006]: 2024-03-24T11:14:25.942+0000 7fe0046e9700 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:14:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:25 smithi100 ceph-mon[28941]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-03-24T11:14:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:25 smithi100 ceph-mon[28941]: osdmap e25: 4 total, 3 up, 4 in 2024-03-24T11:14:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:25 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:25 smithi100 ceph-mon[28941]: from='osd.3 [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:25 smithi100 ceph-mon[28941]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:25 smithi100 ceph-mon[28941]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 875 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:27.229 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:26 smithi105 ceph-mon[36437]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-03-24T11:14:27.230 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:26 smithi105 ceph-mon[36437]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T11:14:27.230 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:26 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:27.230 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:26 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:27.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:26 smithi100 ceph-mon[28941]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-03-24T11:14:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:26 smithi100 ceph-mon[28941]: osdmap e26: 4 total, 3 up, 4 in 2024-03-24T11:14:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:26 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:27.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:26 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:27.885 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-03-24T11:14:27.886 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-03-24T11:14:27.886 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-03-24T11:14:27.886 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-03-24T11:14:27.886 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0130978 s, 801 MB/s 2024-03-24T11:14:27.886 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-03-24T11:14:28.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: purged_snaps scrub starts 2024-03-24T11:14:28.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: purged_snaps scrub ok 2024-03-24T11:14:28.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:28.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: osd.3 [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] boot 2024-03-24T11:14:28.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:14:28.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:28.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:28.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:27 smithi105 ceph-mon[36437]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T11:14:28.294 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: purged_snaps scrub starts 2024-03-24T11:14:28.294 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: purged_snaps scrub ok 2024-03-24T11:14:28.294 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:28.294 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: osd.3 [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] boot 2024-03-24T11:14:28.294 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: osdmap e27: 4 total, 4 up, 4 in 2024-03-24T11:14:28.295 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-03-24T11:14:28.295 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: pgmap v92: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-03-24T11:14:28.295 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:27 smithi100 ceph-mon[28941]: osdmap e28: 4 total, 4 up, 4 in 2024-03-24T11:14:28.511 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-03-24T11:14:28.762 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: Detected new or changed devices on smithi105 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: Adjusting osd_memory_target on smithi105 to 15936M 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:30.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:30.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:30.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:30.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:30.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:29 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.164 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: osdmap e29: 4 total, 4 up, 4 in 2024-03-24T11:14:30.164 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: pgmap v95: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:30.164 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: Detected new or changed devices on smithi105 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: Adjusting osd_memory_target on smithi105 to 15936M 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:30.165 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:29 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 -- 172.21.15.105:0/3721148149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fe440 msgr2=0x7fe0480fe850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 --2- 172.21.15.105:0/3721148149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fe440 0x7fe0480fe850 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fe038004940 tx=0x7fe038009e30 comp rx=0 tx=0).stop 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 -- 172.21.15.105:0/3721148149 shutdown_connections 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 --2- 172.21.15.105:0/3721148149 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480fed90 0x7fe0480faed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 --2- 172.21.15.105:0/3721148149 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fe440 0x7fe0480fe850 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 -- 172.21.15.105:0/3721148149 >> 172.21.15.105:0/3721148149 conn(0x7fe0480f6be0 msgr2=0x7fe0480f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 -- 172.21.15.105:0/3721148149 shutdown_connections 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 -- 172.21.15.105:0/3721148149 wait complete. 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 Processor -- start 2024-03-24T11:14:30.628 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.626+0000 7fe04d796700 1 -- start start 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe04d796700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480fe440 0x7fe04810d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe04d796700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 0x7fe04810de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe04d796700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04810e470 con 0x7fe0480fe440 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe04d796700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe04810e5e0 con 0x7fe0480fed90 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 0x7fe04810de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 0x7fe04810de70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57598/0 (socket says 172.21.15.105:57598) 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 -- 172.21.15.105:0/3707666334 learned_addr learned my addr 172.21.15.105:0/3707666334 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe046ffd700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480fe440 0x7fe04810d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 -- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480fe440 msgr2=0x7fe04810d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480fe440 0x7fe04810d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 -- 172.21.15.105:0/3707666334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe038005040 con 0x7fe0480fed90 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03e7fc700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 0x7fe04810de70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe030002a50 tx=0x7fe0300046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:14:30.629 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe030007cc0 con 0x7fe0480fed90 2024-03-24T11:14:30.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe03000f040 con 0x7fe0480fed90 2024-03-24T11:14:30.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe03000da70 con 0x7fe0480fed90 2024-03-24T11:14:30.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.627+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe048113040 con 0x7fe0480fed90 2024-03-24T11:14:30.630 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.628+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0481135d0 con 0x7fe0480fed90 2024-03-24T11:14:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.629+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe04804c710 con 0x7fe0480fed90 2024-03-24T11:14:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.629+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe03000dc10 con 0x7fe0480fed90 2024-03-24T11:14:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.629+0000 7fe03ffff700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0340692a0 0x7fe03406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.629+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 3770+0+0 (secure 0 0 0) 0x7fe030086140 con 0x7fe0480fed90 2024-03-24T11:14:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.630+0000 7fe046ffd700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0340692a0 0x7fe03406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:30.631 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.630+0000 7fe046ffd700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0340692a0 0x7fe03406b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe038004890 tx=0x7fe038009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:14:30.634 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.634+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe03004d190 con 0x7fe0480fed90 2024-03-24T11:14:30.831 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:30.831+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fe048102490 con 0x7fe0340692a0 2024-03-24T11:14:31.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:30 smithi100 ceph-mon[28941]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T11:14:31.236 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:30 smithi105 ceph-mon[36437]: osdmap e30: 4 total, 4 up, 4 in 2024-03-24T11:14:32.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:31 smithi100 ceph-mon[28941]: osdmap e31: 4 total, 4 up, 4 in 2024-03-24T11:14:32.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:31 smithi100 ceph-mon[28941]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:32.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:31 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:14:32.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:31 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:14:32.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:31 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:32.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:31 smithi100 ceph-mon[28941]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:14:32.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:31 smithi105 ceph-mon[36437]: osdmap e31: 4 total, 4 up, 4 in 2024-03-24T11:14:32.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:31 smithi105 ceph-mon[36437]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:32.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:31 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:14:32.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:31 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:14:32.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:31 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:32.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:31 smithi105 ceph-mon[36437]: pgmap v98: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 144 KiB/s, 0 objects/s recovering 2024-03-24T11:14:33.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:33 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/397533084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c59e7571-57a0-406f-bdbb-614deeaaa56e"}]: dispatch 2024-03-24T11:14:33.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:33 smithi105 ceph-mon[36437]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c59e7571-57a0-406f-bdbb-614deeaaa56e"}]: dispatch 2024-03-24T11:14:33.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:33 smithi105 ceph-mon[36437]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c59e7571-57a0-406f-bdbb-614deeaaa56e"}]': finished 2024-03-24T11:14:33.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:33 smithi105 ceph-mon[36437]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T11:14:33.514 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:33 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:33.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:33 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/397533084' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c59e7571-57a0-406f-bdbb-614deeaaa56e"}]: dispatch 2024-03-24T11:14:33.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:33 smithi100 ceph-mon[28941]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "c59e7571-57a0-406f-bdbb-614deeaaa56e"}]: dispatch 2024-03-24T11:14:33.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:33 smithi100 ceph-mon[28941]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "c59e7571-57a0-406f-bdbb-614deeaaa56e"}]': finished 2024-03-24T11:14:33.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:33 smithi100 ceph-mon[28941]: osdmap e32: 5 total, 4 up, 5 in 2024-03-24T11:14:33.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:33 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:34.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:34 smithi105 ceph-mon[36437]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-24T11:14:34.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:34 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/848322208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:14:34.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:34 smithi100 ceph-mon[28941]: pgmap v100: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 133 KiB/s, 0 objects/s recovering 2024-03-24T11:14:34.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:34 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/848322208' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:14:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:36 smithi105 ceph-mon[36437]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:14:36.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:36 smithi100 ceph-mon[28941]: pgmap v101: 1 pgs: 1 peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-03-24T11:14:38.344 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:38 smithi105 ceph-mon[36437]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T11:14:38.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:38 smithi100 ceph-mon[28941]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 79 KiB/s, 0 objects/s recovering 2024-03-24T11:14:39.356 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:39 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:39.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:39 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:40.304 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:40 smithi105 ceph-mon[36437]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:40.447 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:40 smithi100 ceph-mon[28941]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:41.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:41 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:14:41.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:41 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:41.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:41 smithi105 ceph-mon[36437]: Deploying daemon osd.4 on smithi105 2024-03-24T11:14:41.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:41 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-03-24T11:14:41.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:41 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:41.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:41 smithi100 ceph-mon[28941]: Deploying daemon osd.4 on smithi105 2024-03-24T11:14:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:42 smithi105 ceph-mon[36437]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:42.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:42 smithi100 ceph-mon[28941]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:44.499 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:44 smithi105 ceph-mon[36437]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:44.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:44 smithi100 ceph-mon[28941]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:46.409 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:46 smithi105 ceph-mon[36437]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:46.409 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:46 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:46.409 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:46 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:46.409 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:46 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:46.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:46 smithi100 ceph-mon[28941]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:46.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:46 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:46.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:46 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:46.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:46 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:48 smithi105 ceph-mon[36437]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:48.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:48 smithi100 ceph-mon[28941]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:49.349 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 4 on host 'smithi105' 2024-03-24T11:14:49.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.348+0000 7fe03ffff700 1 -- 172.21.15.105:0/3707666334 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fe048102490 con 0x7fe0340692a0 2024-03-24T11:14:49.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0340692a0 msgr2=0x7fe03406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0340692a0 0x7fe03406b750 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7fe038004890 tx=0x7fe038009e30 comp rx=0 tx=0).stop 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 msgr2=0x7fe04810de70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 0x7fe04810de70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fe030002a50 tx=0x7fe0300046b0 comp rx=0 tx=0).stop 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 shutdown_connections 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0340692a0 0x7fe03406b750 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe0480fe440 0x7fe04810d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 --2- 172.21.15.105:0/3707666334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe0480fed90 0x7fe04810de70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 >> 172.21.15.105:0/3707666334 conn(0x7fe0480f6be0 msgr2=0x7fe0480f8260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 shutdown_connections 2024-03-24T11:14:49.351 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:49.350+0000 7fe04d796700 1 -- 172.21.15.105:0/3707666334 wait complete. 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='osd.4 [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:49.926 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='osd.4 [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:49.927 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:14:49.956 DEBUG:teuthology.orchestra.run.smithi105:osd.4> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.4.service 2024-03-24T11:14:49.958 INFO:tasks.cephadm:Deploying osd.5 on smithi105 with /dev/vg_nvme/lv_2... 2024-03-24T11:14:49.958 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-03-24T11:14:50.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:50 smithi100 ceph-mon[28941]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:50.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:50 smithi100 ceph-mon[28941]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:14:50.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:50 smithi100 ceph-mon[28941]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T11:14:50.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:50.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:50 smithi100 ceph-mon[28941]: from='osd.4 [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:50.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:50 smithi100 ceph-mon[28941]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:50.910 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-mon[36437]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:50.910 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-mon[36437]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-03-24T11:14:50.910 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-mon[36437]: osdmap e33: 5 total, 4 up, 5 in 2024-03-24T11:14:50.910 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:50.910 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-mon[36437]: from='osd.4 [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:50.911 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-mon[36437]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:14:50.911 INFO:journalctl@ceph.osd.4.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-4[48720]: 2024-03-24T11:14:50.645+0000 7f8b381d0700 -1 osd.4 0 waiting for initial osdmap 2024-03-24T11:14:50.911 INFO:journalctl@ceph.osd.4.smithi105.stdout:Mar 24 11:14:50 smithi105 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-4[48720]: 2024-03-24T11:14:50.652+0000 7f8b2f7c4700 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:14:50.964 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:14:51.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-03-24T11:14:51.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: osdmap e34: 5 total, 4 up, 5 in 2024-03-24T11:14:51.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:51.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:51.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: osd.4 [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] boot 2024-03-24T11:14:51.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:14:51.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:51 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:52.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-03-24T11:14:52.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: osdmap e34: 5 total, 4 up, 5 in 2024-03-24T11:14:52.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:52.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:52.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: osd.4 [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] boot 2024-03-24T11:14:52.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: osdmap e35: 5 total, 5 up, 5 in 2024-03-24T11:14:52.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:51 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-03-24T11:14:53.033 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:52 smithi105 ceph-mon[36437]: purged_snaps scrub starts 2024-03-24T11:14:53.033 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:52 smithi105 ceph-mon[36437]: purged_snaps scrub ok 2024-03-24T11:14:53.033 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:52 smithi105 ceph-mon[36437]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:53.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:52 smithi100 ceph-mon[28941]: purged_snaps scrub starts 2024-03-24T11:14:53.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:52 smithi100 ceph-mon[28941]: purged_snaps scrub ok 2024-03-24T11:14:53.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:52 smithi100 ceph-mon[28941]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-03-24T11:14:53.709 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-03-24T11:14:53.710 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-03-24T11:14:53.710 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-03-24T11:14:53.710 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-03-24T11:14:53.710 INFO:teuthology.orchestra.run.smithi105.stderr:10485760 bytes (10 MB, 10 MiB) copied, 0.0116074 s, 903 MB/s 2024-03-24T11:14:53.710 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-03-24T11:14:54.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T11:14:54.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:54.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:54.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:54.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:54.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: osdmap e36: 5 total, 5 up, 5 in 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:14:54.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:14:54.446 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-03-24T11:14:54.699 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:14:54.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:54 smithi105 ceph-mon[36437]: Detected new or changed devices on smithi105 2024-03-24T11:14:54.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:54 smithi105 ceph-mon[36437]: Adjusting osd_memory_target on smithi105 to 7968M 2024-03-24T11:14:54.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:54 smithi105 ceph-mon[36437]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:14:55.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:54 smithi100 ceph-mon[28941]: Detected new or changed devices on smithi105 2024-03-24T11:14:55.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:54 smithi100 ceph-mon[28941]: Adjusting osd_memory_target on smithi105 to 7968M 2024-03-24T11:14:55.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:54 smithi100 ceph-mon[28941]: pgmap v114: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 -- 172.21.15.105:0/285858551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fdfe0 msgr2=0x7f6f8c0fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/285858551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fdfe0 0x7f6f8c0fe450 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7f6f78004940 tx=0x7f6f78009e30 comp rx=0 tx=0).stop 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 -- 172.21.15.105:0/285858551 shutdown_connections 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/285858551 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fdfe0 0x7f6f8c0fe450 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/285858551 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c0fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 -- 172.21.15.105:0/285858551 >> 172.21.15.105:0/285858551 conn(0x7f6f8c0f8550 msgr2=0x7f6f8c0fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 -- 172.21.15.105:0/285858551 shutdown_connections 2024-03-24T11:14:56.218 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 -- 172.21.15.105:0/285858551 wait complete. 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 Processor -- start 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.215+0000 7f6f92eb7700 1 -- start start 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f92eb7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c10d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f92eb7700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f8c0fdfe0 0x7f6f8c10de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f92eb7700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c10e460 con 0x7f6f8c0fdfe0 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f92eb7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f8c10e5d0 con 0x7f6f8c0fce70 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c10d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c10d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37494/0 (socket says 172.21.15.105:37494) 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 -- 172.21.15.105:0/518237378 learned_addr learned my addr 172.21.15.105:0/518237378 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f8bfff700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f8c0fdfe0 0x7f6f8c10de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 -- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f8c0fdfe0 msgr2=0x7f6f8c10de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f8c0fdfe0 0x7f6f8c10de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:14:56.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 -- 172.21.15.105:0/518237378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f78005040 con 0x7f6f8c0fce70 2024-03-24T11:14:56.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f90c53700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c10d920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6f80002a50 tx=0x7f6f800046b0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:14:56.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f80007cc0 con 0x7f6f8c0fce70 2024-03-24T11:14:56.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f8c112fd0 con 0x7f6f8c0fce70 2024-03-24T11:14:56.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6f8000f040 con 0x7f6f8c0fce70 2024-03-24T11:14:56.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.216+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6f8000da30 con 0x7f6f8c0fce70 2024-03-24T11:14:56.220 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.217+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f8c113560 con 0x7f6f8c0fce70 2024-03-24T11:14:56.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.217+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f8c04c710 con 0x7f6f8c0fce70 2024-03-24T11:14:56.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.218+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6f8000db90 con 0x7f6f8c0fce70 2024-03-24T11:14:56.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.218+0000 7f6f897fa700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6f7c0692a0 0x7f6f7c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:14:56.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.218+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7f6f80086a50 con 0x7f6f8c0fce70 2024-03-24T11:14:56.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.218+0000 7f6f8bfff700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6f7c0692a0 0x7f6f7c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:14:56.221 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.219+0000 7f6f8bfff700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6f7c0692a0 0x7f6f7c06b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800b970 tx=0x7f6f78009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:14:56.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.223+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6f80056c80 con 0x7f6f8c0fce70 2024-03-24T11:14:56.418 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:14:56.417+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f8c113750 con 0x7f6f7c0692a0 2024-03-24T11:14:56.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:56 smithi105 ceph-mon[36437]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:14:56.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:14:56.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:14:56.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:57.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:56 smithi100 ceph-mon[28941]: pgmap v115: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:14:57.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-03-24T11:14:57.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-03-24T11:14:57.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:14:57.914 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:57 smithi105 ceph-mon[36437]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:58.044 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:57 smithi100 ceph-mon[28941]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:14:58.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:58 smithi100 ceph-mon[28941]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:14:59.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:58 smithi105 ceph-mon[36437]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:00.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:59 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/596365098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b5ed12-6a0d-4bfe-9ffd-5224e21353ae"}]: dispatch 2024-03-24T11:15:00.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:59 smithi105 ceph-mon[36437]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b5ed12-6a0d-4bfe-9ffd-5224e21353ae"}]: dispatch 2024-03-24T11:15:00.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:59 smithi105 ceph-mon[36437]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15b5ed12-6a0d-4bfe-9ffd-5224e21353ae"}]': finished 2024-03-24T11:15:00.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:59 smithi105 ceph-mon[36437]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:15:00.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:59 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:00.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:14:59 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/3377961313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:59 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/596365098' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b5ed12-6a0d-4bfe-9ffd-5224e21353ae"}]: dispatch 2024-03-24T11:15:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:59 smithi100 ceph-mon[28941]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "15b5ed12-6a0d-4bfe-9ffd-5224e21353ae"}]: dispatch 2024-03-24T11:15:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:59 smithi100 ceph-mon[28941]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "15b5ed12-6a0d-4bfe-9ffd-5224e21353ae"}]': finished 2024-03-24T11:15:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:59 smithi100 ceph-mon[28941]: osdmap e37: 6 total, 5 up, 6 in 2024-03-24T11:15:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:59 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:00.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:14:59 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/3377961313' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-03-24T11:15:01.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:00 smithi105 ceph-mon[36437]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:01.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:00 smithi100 ceph-mon[28941]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:03.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:02 smithi105 ceph-mon[36437]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:03.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:02 smithi100 ceph-mon[28941]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:04.962 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:04 smithi105 ceph-mon[36437]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:05.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:04 smithi100 ceph-mon[28941]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:06.881 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:06 smithi105 ceph-mon[36437]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:06.881 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:15:06.881 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:07.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:06 smithi100 ceph-mon[28941]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:07.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-03-24T11:15:07.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:07 smithi105 ceph-mon[36437]: Deploying daemon osd.5 on smithi105 2024-03-24T11:15:08.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:07 smithi100 ceph-mon[28941]: Deploying daemon osd.5 on smithi105 2024-03-24T11:15:08.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:08 smithi105 ceph-mon[36437]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:08.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:08.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:08 smithi100 ceph-mon[28941]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:08.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:11.053 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:10 smithi105 ceph-mon[36437]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:11.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:10 smithi100 ceph-mon[28941]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:12.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:12 smithi105 ceph-mon[36437]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:12.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:12 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:12.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:12 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:12.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:12 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:12.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:12 smithi100 ceph-mon[28941]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:12.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:12 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:12.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:12 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:12.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:12 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:14.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:14 smithi105 ceph-mon[36437]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:14.882 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.882+0000 7f6f897fa700 1 -- 172.21.15.105:0/518237378 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f6f8c113750 con 0x7f6f7c0692a0 2024-03-24T11:15:14.883 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 5 on host 'smithi105' 2024-03-24T11:15:14.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.884+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6f7c0692a0 msgr2=0x7f6f7c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:14.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.884+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6f7c0692a0 0x7f6f7c06b750 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f6f7800b970 tx=0x7f6f78009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:14.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.884+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 msgr2=0x7f6f8c10d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:14.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.884+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c10d920 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7f6f80002a50 tx=0x7f6f800046b0 comp rx=0 tx=0).stop 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 shutdown_connections 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6f7c0692a0 0x7f6f7c06b750 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6f8c0fce70 0x7f6f8c10d920 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 --2- 172.21.15.105:0/518237378 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6f8c0fdfe0 0x7f6f8c10de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 >> 172.21.15.105:0/518237378 conn(0x7f6f8c0f8550 msgr2=0x7f6f8c0f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 shutdown_connections 2024-03-24T11:15:14.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:14.885+0000 7f6f92eb7700 1 -- 172.21.15.105:0/518237378 wait complete. 2024-03-24T11:15:14.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:14 smithi100 ceph-mon[28941]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:15.833 DEBUG:teuthology.orchestra.run.smithi105:osd.5> sudo journalctl -f -n 0 -u ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.5.service 2024-03-24T11:15:15.836 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-03-24T11:15:15.836 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd stat -f json 2024-03-24T11:15:16.105 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:16.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:15 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:16.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:15 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:16.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:15 smithi100 ceph-mon[28941]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:16.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:15 smithi100 ceph-mon[28941]: from='osd.5 [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:15:16.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:15 smithi100 ceph-mon[28941]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:15:16.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:16.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:15 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:16.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:15 smithi105 ceph-mon[36437]: pgmap v126: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:16.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:15 smithi105 ceph-mon[36437]: from='osd.5 [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:15:16.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:15 smithi105 ceph-mon[36437]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: from='osd.5 [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:17.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:16 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: osdmap e38: 6 total, 5 up, 6 in 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: from='osd.5 [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:17.146 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:17.146 INFO:journalctl@ceph.osd.5.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-5[55618]: 2024-03-24T11:15:16.865+0000 7f2d05cbd700 -1 osd.5 0 waiting for initial osdmap 2024-03-24T11:15:17.146 INFO:journalctl@ceph.osd.5.smithi105.stdout:Mar 24 11:15:16 smithi105 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-5[55618]: 2024-03-24T11:15:16.877+0000 7f2cffab6700 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 -- 172.21.15.100:0/3335857594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 msgr2=0x7fc2801023a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 --2- 172.21.15.100:0/3335857594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc2801023a0 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 crypto rx=0x7fc270004940 tx=0x7fc270009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 -- 172.21.15.100:0/3335857594 shutdown_connections 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 --2- 172.21.15.100:0/3335857594 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc2801023a0 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 --2- 172.21.15.100:0/3335857594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc280066220 0x7fc2800ffa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 -- 172.21.15.100:0/3335857594 >> 172.21.15.100:0/3335857594 conn(0x7fc2800f7430 msgr2=0x7fc2800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 -- 172.21.15.100:0/3335857594 shutdown_connections 2024-03-24T11:15:17.939 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 -- 172.21.15.100:0/3335857594 wait complete. 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 Processor -- start 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.937+0000 7fc287820700 1 -- start start 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc287820700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc280066220 0x7fc28010da40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc287820700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc28010df80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc287820700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28010e580 con 0x7fc2800fffc0 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc287820700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc28010ed80 con 0x7fc280066220 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc28010df80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc28010df80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:34164/0 (socket says 172.21.15.100:34164) 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 -- 172.21.15.100:0/3348318237 learned_addr learned my addr 172.21.15.100:0/3348318237 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc2855bc700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc280066220 0x7fc28010da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 -- 172.21.15.100:0/3348318237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc280066220 msgr2=0x7fc28010da40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc280066220 0x7fc28010da40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:17.940 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 -- 172.21.15.100:0/3348318237 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc270005040 con 0x7fc2800fffc0 2024-03-24T11:15:17.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc284dbb700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc28010df80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc270000c00 tx=0x7fc270009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:17.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc27000e030 con 0x7fc2800fffc0 2024-03-24T11:15:17.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc270013de0 con 0x7fc2800fffc0 2024-03-24T11:15:17.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc27001ce50 con 0x7fc2800fffc0 2024-03-24T11:15:17.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.938+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc28010f000 con 0x7fc2800fffc0 2024-03-24T11:15:17.941 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.939+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc28010f4f0 con 0x7fc2800fffc0 2024-03-24T11:15:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.940+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc28005f5f0 con 0x7fc2800fffc0 2024-03-24T11:15:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.940+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc27001c400 con 0x7fc2800fffc0 2024-03-24T11:15:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.940+0000 7fc275ffb700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc26c0692a0 0x7fc26c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.940+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc270088840 con 0x7fc2800fffc0 2024-03-24T11:15:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.941+0000 7fc2855bc700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc26c0692a0 0x7fc26c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:17.942 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.941+0000 7fc2855bc700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc26c0692a0 0x7fc26c06b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc280065f60 tx=0x7fc27c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:17.945 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:17.945+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc270058940 con 0x7fc2800fffc0 2024-03-24T11:15:18.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:17 smithi105 ceph-mon[36437]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-03-24T11:15:18.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:17 smithi105 ceph-mon[36437]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:15:18.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:17 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:18.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:17 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:18.052 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:17 smithi105 ceph-mon[36437]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:18.132 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.131+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fc28004bed0 con 0x7fc2800fffc0 2024-03-24T11:15:18.132 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:17 smithi100 ceph-mon[28941]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-03-24T11:15:18.132 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:17 smithi100 ceph-mon[28941]: osdmap e39: 6 total, 5 up, 6 in 2024-03-24T11:15:18.132 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:17 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:18.132 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:17 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:18.132 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:17 smithi100 ceph-mon[28941]: pgmap v129: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-03-24T11:15:18.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.131+0000 7fc275ffb700 1 -- 172.21.15.100:0/3348318237 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v40) v1 ==== 74+0+130 (secure 0 0 0) 0x7fc2700584d0 con 0x7fc2800fffc0 2024-03-24T11:15:18.133 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:15:18.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc26c0692a0 msgr2=0x7fc26c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:18.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc26c0692a0 0x7fc26c06b750 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fc280065f60 tx=0x7fc27c00b040 comp rx=0 tx=0).stop 2024-03-24T11:15:18.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 msgr2=0x7fc28010df80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:18.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc28010df80 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fc270000c00 tx=0x7fc270009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:18.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 shutdown_connections 2024-03-24T11:15:18.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc26c0692a0 0x7fc26c06b750 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:18.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc280066220 0x7fc28010da40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:18.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 --2- 172.21.15.100:0/3348318237 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2800fffc0 0x7fc28010df80 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:18.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 >> 172.21.15.100:0/3348318237 conn(0x7fc2800f7430 msgr2=0x7fc2800f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:18.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 shutdown_connections 2024-03-24T11:15:18.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:18.134+0000 7fc287820700 1 -- 172.21.15.100:0/3348318237 wait complete. 2024-03-24T11:15:18.775 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":40,"num_osds":6,"num_up_osds":6,"osd_up_since":1711278917,"num_in_osds":6,"osd_in_since":1711278898,"num_remapped_pgs":0} 2024-03-24T11:15:18.775 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:15:19.041 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:19.106 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:18 smithi105 ceph-mon[36437]: purged_snaps scrub starts 2024-03-24T11:15:19.106 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:18 smithi105 ceph-mon[36437]: purged_snaps scrub ok 2024-03-24T11:15:19.106 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:18 smithi105 ceph-mon[36437]: osd.5 [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] boot 2024-03-24T11:15:19.106 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:18 smithi105 ceph-mon[36437]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:15:19.106 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:18 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:19.106 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:18 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3348318237' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:15:19.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:18 smithi100 ceph-mon[28941]: purged_snaps scrub starts 2024-03-24T11:15:19.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:18 smithi100 ceph-mon[28941]: purged_snaps scrub ok 2024-03-24T11:15:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:18 smithi100 ceph-mon[28941]: osd.5 [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] boot 2024-03-24T11:15:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:18 smithi100 ceph-mon[28941]: osdmap e40: 6 total, 6 up, 6 in 2024-03-24T11:15:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:18 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-03-24T11:15:19.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:18 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3348318237' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: Detected new or changed devices on smithi105 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: Adjusting osd_memory_target on smithi105 to 5312M 2024-03-24T11:15:20.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:20.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:20.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: osdmap e41: 6 total, 6 up, 6 in 2024-03-24T11:15:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: Detected new or changed devices on smithi105 2024-03-24T11:15:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: Adjusting osd_memory_target on smithi105 to 5312M 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:15:21.042 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 -- 172.21.15.100:0/2443855707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 msgr2=0x7fed780fb990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:21.043 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/2443855707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed780fb990 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fed64004940 tx=0x7fed64009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:21.043 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.043 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:21.043 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:20 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 -- 172.21.15.100:0/2443855707 shutdown_connections 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/2443855707 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed780fb990 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/2443855707 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 0x7fed781033a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 -- 172.21.15.100:0/2443855707 >> 172.21.15.100:0/2443855707 conn(0x7fed78071530 msgr2=0x7fed78071930 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.041+0000 7fed7ebe5700 1 -- 172.21.15.100:0/2443855707 shutdown_connections 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 -- 172.21.15.100:0/2443855707 wait complete. 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 Processor -- start 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 -- start start 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed78113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 0x7fed78113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed781144c0 con 0x7fed780fb540 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.042+0000 7fed7ebe5700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed7810db00 con 0x7fed78102fd0 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed78113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:21.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed78113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:34180/0 (socket says 172.21.15.100:34180) 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 -- 172.21.15.100:0/1185037082 learned_addr learned my addr 172.21.15.100:0/1185037082 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed77fff700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 0x7fed78113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 -- 172.21.15.100:0/1185037082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 msgr2=0x7fed78113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 0x7fed78113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 -- 172.21.15.100:0/1185037082 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed64005040 con 0x7fed780fb540 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed77fff700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 0x7fed78113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7c981700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed78113930 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c810 tx=0x7fed6c004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6c012810 con 0x7fed780fb540 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed7810dde0 con 0x7fed780fb540 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fed6c012e40 con 0x7fed780fb540 2024-03-24T11:15:21.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed6c00a370 con 0x7fed780fb540 2024-03-24T11:15:21.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.043+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed7810e3a0 con 0x7fed780fb540 2024-03-24T11:15:21.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.044+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed781053b0 con 0x7fed780fb540 2024-03-24T11:15:21.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.045+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fed6c00a550 con 0x7fed780fb540 2024-03-24T11:15:21.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.045+0000 7fed757fa700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fed680692a0 0x7fed6806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:21.047 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.045+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fed6c0872f0 con 0x7fed780fb540 2024-03-24T11:15:21.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.045+0000 7fed77fff700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fed680692a0 0x7fed6806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:21.048 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.046+0000 7fed77fff700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fed680692a0 0x7fed6806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fed64009fd0 tx=0x7fed64009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:21.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.050+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fed6c057150 con 0x7fed780fb540 2024-03-24T11:15:21.236 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.236+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed7810a030 con 0x7fed780fb540 2024-03-24T11:15:21.237 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.236+0000 7fed757fa700 1 -- 172.21.15.100:0/1185037082 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11135 (secure 0 0 0) 0x7fed6c056f70 con 0x7fed780fb540 2024-03-24T11:15:21.237 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:15:21.237 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":41,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","created":"2024-03-24T11:09:42.710984+0000","modified":"2024-03-24T11:15:18.868228+0000","last_up_change":"2024-03-24T11:15:17.863076+0000","last_in_change":"2024-03-24T11:14:58.878641+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:14:05.424502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cc6e0f08-af11-46b2-923f-17e549042b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6803","nonce":3447712544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6805","nonce":3447712544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6809","nonce":3447712544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6807","nonce":3447712544}]},"public_addr":"172.21.15.100:6803/3447712544","cluster_addr":"172.21.15.100:6805/3447712544","heartbeat_back_addr":"172.21.15.100:6809/3447712544","heartbeat_front_addr":"172.21.15.100:6807/3447712544","state":["exists","up"]},{"osd":1,"uuid":"3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6811","nonce":1490052239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6813","nonce":1490052239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6817","nonce":1490052239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6815","nonce":1490052239}]},"public_addr":"172.21.15.100:6811/1490052239","cluster_addr":"172.21.15.100:6813/1490052239","heartbeat_back_addr":"172.21.15.100:6817/1490052239","heartbeat_front_addr":"172.21.15.100:6815/1490052239","state":["exists","up"]},{"osd":2,"uuid":"44db3f0c-2877-4026-93f3-febdb7f46d67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6819","nonce":2671131686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6821","nonce":2671131686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6825","nonce":2671131686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6823","nonce":2671131686}]},"public_addr":"172.21.15.100:6819/2671131686","cluster_addr":"172.21.15.100:6821/2671131686","heartbeat_back_addr":"172.21.15.100:6825/2671131686","heartbeat_front_addr":"172.21.15.100:6823/2671131686","state":["exists","up"]},{"osd":3,"uuid":"1f6ededc-bddf-4308-a416-80ae19ad3fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6801","nonce":3221138489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6803","nonce":3221138489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6807","nonce":3221138489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6805","nonce":3221138489}]},"public_addr":"172.21.15.105:6801/3221138489","cluster_addr":"172.21.15.105:6803/3221138489","heartbeat_back_addr":"172.21.15.105:6807/3221138489","heartbeat_front_addr":"172.21.15.105:6805/3221138489","state":["exists","up"]},{"osd":4,"uuid":"c59e7571-57a0-406f-bdbb-614deeaaa56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6809","nonce":3756920843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6811","nonce":3756920843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6815","nonce":3756920843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6813","nonce":3756920843}]},"public_addr":"172.21.15.105:6809/3756920843","cluster_addr":"172.21.15.105:6811/3756920843","heartbeat_back_addr":"172.21.15.105:6815/3756920843","heartbeat_front_addr":"172.21.15.105:6813/3756920843","state":["exists","up"]},{"osd":5,"uuid":"15b5ed12-6a0d-4bfe-9ffd-5224e21353ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6817","nonce":256673904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6819","nonce":256673904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6823","nonce":256673904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6821","nonce":256673904}]},"public_addr":"172.21.15.105:6817/256673904","cluster_addr":"172.21.15.105:6819/256673904","heartbeat_back_addr":"172.21.15.105:6823/256673904","heartbeat_front_addr":"172.21.15.105:6821/256673904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:03.013039+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:32.779377+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:01.889286+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:25.258963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:50.202419+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:15:16.423165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/2476598760":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/3850505170":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6801/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/374004208":"2024-03-25T11:10:47.648919+0000","172.21.15.100:6801/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/3084685173":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/1681068182":"2024-03-25T11:10:16.917673+0000","172.21.15.100:0/3561410096":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2555446076":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6801/1648043756":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6800/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2493584253":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/4249885134":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1648043756":"2024-03-25T11:10:16.917673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fed680692a0 msgr2=0x7fed6806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fed680692a0 0x7fed6806b750 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fed64009fd0 tx=0x7fed64009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 msgr2=0x7fed78113930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed78113930 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7fed6c00c810 tx=0x7fed6c004470 comp rx=0 tx=0).stop 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 shutdown_connections 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fed680692a0 0x7fed6806b750 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fed780fb540 0x7fed78113930 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 --2- 172.21.15.100:0/1185037082 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed78102fd0 0x7fed78113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.238+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 >> 172.21.15.100:0/1185037082 conn(0x7fed78071530 msgr2=0x7fed780fa780 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:21.239 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.239+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 shutdown_connections 2024-03-24T11:15:21.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:21.239+0000 7fed7ebe5700 1 -- 172.21.15.100:0/1185037082 wait complete. 2024-03-24T11:15:21.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:15:21.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:15:21.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:20 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:15:21.949 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:14:05.424502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-03-24T11:15:21.949 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd pool get .mgr pg_num 2024-03-24T11:15:22.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:21 smithi100 ceph-mon[28941]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:22.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:21 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1185037082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:15:22.206 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:22.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:21 smithi105 ceph-mon[36437]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:22.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:21 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/1185037082' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:15:23.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:23.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:24.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 -- 172.21.15.100:0/3854350008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec01026c0 msgr2=0x7fbec00fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:24.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 --2- 172.21.15.100:0/3854350008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec01026c0 0x7fbec00fbd30 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7fbeb4004940 tx=0x7fbeb4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:24.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 -- 172.21.15.100:0/3854350008 shutdown_connections 2024-03-24T11:15:24.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 --2- 172.21.15.100:0/3854350008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec01026c0 0x7fbec00fbd30 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:24.045 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 --2- 172.21.15.100:0/3854350008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec0101d20 0x7fbec01020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:24.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 -- 172.21.15.100:0/3854350008 >> 172.21.15.100:0/3854350008 conn(0x7fbec00f7870 msgr2=0x7fbec00f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:24.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 -- 172.21.15.100:0/3854350008 shutdown_connections 2024-03-24T11:15:24.046 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.044+0000 7fbec5724700 1 -- 172.21.15.100:0/3854350008 wait complete. 2024-03-24T11:15:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.045+0000 7fbec5724700 1 Processor -- start 2024-03-24T11:15:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.045+0000 7fbec5724700 1 -- start start 2024-03-24T11:15:24.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.045+0000 7fbec5724700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 0x7fbec01128a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbec5724700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec01026c0 0x7fbec0112de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbec5724700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec01134c0 con 0x7fbec0101d20 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbec5724700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbec010ca30 con 0x7fbec01026c0 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 0x7fbec01128a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 0x7fbec01128a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:34186/0 (socket says 172.21.15.100:34186) 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 -- 172.21.15.100:0/1293700679 learned_addr learned my addr 172.21.15.100:0/1293700679 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebe7fc700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec01026c0 0x7fbec0112de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 -- 172.21.15.100:0/1293700679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec01026c0 msgr2=0x7fbec0112de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec01026c0 0x7fbec0112de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 -- 172.21.15.100:0/1293700679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb4005040 con 0x7fbec0101d20 2024-03-24T11:15:24.051 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebe7fc700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec01026c0 0x7fbec0112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbebeffd700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 0x7fbec01128a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000b790 tx=0x7fbeb0009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb000f770 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbec010cd10 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fbeb000fda0 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbeb000d310 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.046+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbec010d2d0 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.047+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbec00fd470 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.048+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fbeb000f8d0 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.048+0000 7fbea77fe700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fbea80692a0 0x7fbea806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.048+0000 7fbebe7fc700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fbea80692a0 0x7fbea806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.048+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbeb0086ed0 con 0x7fbec0101d20 2024-03-24T11:15:24.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.049+0000 7fbebe7fc700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fbea80692a0 0x7fbea806b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbec010dbf0 tx=0x7fbeb4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:24.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.053+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fbeb0056fd0 con 0x7fbec0101d20 2024-03-24T11:15:24.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.237+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7fbec0108e70 con 0x7fbec0101d20 2024-03-24T11:15:24.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.237+0000 7fbea77fe700 1 -- 172.21.15.100:0/1293700679 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7fbeb0056b60 con 0x7fbec0101d20 2024-03-24T11:15:24.238 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 1 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fbea80692a0 msgr2=0x7fbea806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fbea80692a0 0x7fbea806b750 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fbec010dbf0 tx=0x7fbeb4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 msgr2=0x7fbec01128a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 0x7fbec01128a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7fbeb000b790 tx=0x7fbeb0009250 comp rx=0 tx=0).stop 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 shutdown_connections 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fbea80692a0 0x7fbea806b750 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fbec0101d20 0x7fbec01128a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:24.240 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 --2- 172.21.15.100:0/1293700679 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbec01026c0 0x7fbec0112de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:24.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.239+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 >> 172.21.15.100:0/1293700679 conn(0x7fbec00f7870 msgr2=0x7fbec00f8890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:24.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.240+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 shutdown_connections 2024-03-24T11:15:24.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:24.240+0000 7fbec5724700 1 -- 172.21.15.100:0/1293700679 wait complete. 2024-03-24T11:15:24.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:24 smithi100 ceph-mon[28941]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:24.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:24 smithi105 ceph-mon[36437]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:24.922 INFO:tasks.cephadm:Setting up client nodes... 2024-03-24T11:15:24.922 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:15:25.175 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:25.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:25 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/1293700679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:15:25.576 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:25 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1293700679' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:15:26.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:26 smithi100 ceph-mon[28941]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:26.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:26 smithi105 ceph-mon[36437]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:26.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 -- 172.21.15.100:0/1492777065 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fcfb0 msgr2=0x7f94280fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:26.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 --2- 172.21.15.100:0/1492777065 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fcfb0 0x7f94280fd420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f941c004940 tx=0x7f941c009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:26.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 -- 172.21.15.100:0/1492777065 shutdown_connections 2024-03-24T11:15:26.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 --2- 172.21.15.100:0/1492777065 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fcfb0 0x7f94280fd420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:26.982 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 --2- 172.21.15.100:0/1492777065 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fbd00 0x7f94280fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:26.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 -- 172.21.15.100:0/1492777065 >> 172.21.15.100:0/1492777065 conn(0x7f94280f74a0 msgr2=0x7f94280f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:26.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.981+0000 7f942d17e700 1 -- 172.21.15.100:0/1492777065 shutdown_connections 2024-03-24T11:15:26.983 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.982+0000 7f942d17e700 1 -- 172.21.15.100:0/1492777065 wait complete. 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.982+0000 7f942d17e700 1 Processor -- start 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.982+0000 7f942d17e700 1 -- start start 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.982+0000 7f942d17e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 0x7f9428066d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.982+0000 7f942d17e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fcfb0 0x7f9428114b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.982+0000 7f942d17e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942810e510 con 0x7f94280fbd00 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f942d17e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9428115070 con 0x7f94280fcfb0 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 0x7f9428066d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 0x7f9428066d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:34192/0 (socket says 172.21.15.100:34192) 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 -- 172.21.15.100:0/3550800690 learned_addr learned my addr 172.21.15.100:0/3550800690 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f942659c700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fcfb0 0x7f9428114b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 -- 172.21.15.100:0/3550800690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fcfb0 msgr2=0x7f9428114b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:26.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fcfb0 0x7f9428114b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 -- 172.21.15.100:0/3550800690 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f941c005040 con 0x7f94280fbd00 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f942659c700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fcfb0 0x7f9428114b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f9426d9d700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 0x7f9428066d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9418002a80 tx=0x7f94180046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9418007cc0 con 0x7f94280fbd00 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f941800f040 con 0x7f94280fbd00 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9418009640 con 0x7f94280fbd00 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9428115270 con 0x7f94280fbd00 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.983+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94281157e0 con 0x7f94280fbd00 2024-03-24T11:15:26.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.984+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f942804c710 con 0x7f94280fbd00 2024-03-24T11:15:26.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.985+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f94180097a0 con 0x7f94280fbd00 2024-03-24T11:15:26.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.985+0000 7f940f7fe700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f94100692a0 0x7f941006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:26.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.985+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9418086b50 con 0x7f94280fbd00 2024-03-24T11:15:26.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.985+0000 7f942659c700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f94100692a0 0x7f941006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:26.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.986+0000 7f942659c700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f94100692a0 0x7f941006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f941c00b3b0 tx=0x7f941c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:26.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:26.990+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f9418056c50 con 0x7f94280fbd00 2024-03-24T11:15:27.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.251+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f9428115a50 con 0x7f94280fbd00 2024-03-24T11:15:27.255 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.255+0000 7f940f7fe700 1 -- 172.21.15.100:0/3550800690 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f94180567e0 con 0x7f94280fbd00 2024-03-24T11:15:27.255 INFO:teuthology.orchestra.run.smithi100.stdout:[client.0] 2024-03-24T11:15:27.255 INFO:teuthology.orchestra.run.smithi100.stdout: key = AQBPCwBmLg4ZDxAAIFCxLQrSVqTRp3qJh9KQ7w== 2024-03-24T11:15:27.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f94100692a0 msgr2=0x7f941006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:27.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f94100692a0 0x7f941006b750 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f941c00b3b0 tx=0x7f941c009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:27.257 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 msgr2=0x7f9428066d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 0x7f9428066d30 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f9418002a80 tx=0x7f94180046b0 comp rx=0 tx=0).stop 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 shutdown_connections 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f94100692a0 0x7f941006b750 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f94280fbd00 0x7f9428066d30 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 --2- 172.21.15.100:0/3550800690 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94280fcfb0 0x7f9428114b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 >> 172.21.15.100:0/3550800690 conn(0x7f94280f74a0 msgr2=0x7f94280f8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 shutdown_connections 2024-03-24T11:15:27.258 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:27.257+0000 7f942d17e700 1 -- 172.21.15.100:0/3550800690 wait complete. 2024-03-24T11:15:28.088 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:15:28.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-03-24T11:15:28.088 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-03-24T11:15:28.131 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-03-24T11:15:28.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:28 smithi100 ceph-mon[28941]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:28 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3550800690' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:15:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:28 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3550800690' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:15:28.401 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi105/config 2024-03-24T11:15:28.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:28 smithi105 ceph-mon[36437]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:28.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:28 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3550800690' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:15:28.524 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:28 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3550800690' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 -- 172.21.15.105:0/1427828121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 msgr2=0x7efe000fd440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 --2- 172.21.15.105:0/1427828121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe000fd440 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7efdf4004940 tx=0x7efdf4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 -- 172.21.15.105:0/1427828121 shutdown_connections 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 --2- 172.21.15.105:0/1427828121 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe000fd440 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 --2- 172.21.15.105:0/1427828121 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe000fbd20 0x7efe000fc130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 -- 172.21.15.105:0/1427828121 >> 172.21.15.105:0/1427828121 conn(0x7efe000f7460 msgr2=0x7efe000f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 -- 172.21.15.105:0/1427828121 shutdown_connections 2024-03-24T11:15:29.981 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 -- 172.21.15.105:0/1427828121 wait complete. 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.977+0000 7efe064ab700 1 Processor -- start 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efe064ab700 1 -- start start 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efe064ab700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe000fbd20 0x7efe0010c7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efe064ab700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe0010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efe064ab700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0010d2e0 con 0x7efe000fbd20 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efe064ab700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe0010d450 con 0x7efe000fcfd0 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe0010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdff7fe700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe0010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34022/0 (socket says 172.21.15.105:34022) 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdff7fe700 1 -- 172.21.15.105:0/1144758942 learned_addr learned my addr 172.21.15.105:0/1144758942 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdfffff700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe000fbd20 0x7efe0010c7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdff7fe700 1 -- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe000fbd20 msgr2=0x7efe0010c7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdff7fe700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe000fbd20 0x7efe0010c7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:29.982 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.978+0000 7efdff7fe700 1 -- 172.21.15.105:0/1144758942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4005040 con 0x7efe000fcfd0 2024-03-24T11:15:29.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.979+0000 7efdff7fe700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe0010cce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efdf400b3b0 tx=0x7efdf4009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:29.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.979+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdf400e040 con 0x7efe000fcfd0 2024-03-24T11:15:29.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.979+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7efdf4004290 con 0x7efe000fcfd0 2024-03-24T11:15:29.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.979+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe00111e50 con 0x7efe000fcfd0 2024-03-24T11:15:29.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.979+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efdf401c3f0 con 0x7efe000fcfd0 2024-03-24T11:15:29.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.979+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe000657e0 con 0x7efe000fcfd0 2024-03-24T11:15:29.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.980+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe0004c710 con 0x7efe000fcfd0 2024-03-24T11:15:29.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.980+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7efdf400bbf0 con 0x7efe000fcfd0 2024-03-24T11:15:29.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.981+0000 7efdfcff9700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7efde80692b0 0x7efde806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:29.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.981+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efdf40880b0 con 0x7efe000fcfd0 2024-03-24T11:15:29.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.981+0000 7efdfffff700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7efde80692b0 0x7efde806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:29.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.981+0000 7efdfffff700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7efde80692b0 0x7efde806b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009fd0 tx=0x7efdf000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:29.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:29.985+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7efdf4058250 con 0x7efe000fcfd0 2024-03-24T11:15:30.242 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.242+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7efe00065c90 con 0x7efe000fcfd0 2024-03-24T11:15:30.248 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.247+0000 7efdfcff9700 1 -- 172.21.15.105:0/1144758942 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7efdf4057de0 con 0x7efe000fcfd0 2024-03-24T11:15:30.249 INFO:teuthology.orchestra.run.smithi105.stdout:[client.1] 2024-03-24T11:15:30.249 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQBSCwBmwzutDhAAdVHoc4TjabJM+5sIINCD3Q== 2024-03-24T11:15:30.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.249+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7efde80692b0 msgr2=0x7efde806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:30.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.249+0000 7efe064ab700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7efde80692b0 0x7efde806b760 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7efdf0009fd0 tx=0x7efdf000b040 comp rx=0 tx=0).stop 2024-03-24T11:15:30.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.249+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 msgr2=0x7efe0010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.249+0000 7efe064ab700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe0010cce0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7efdf400b3b0 tx=0x7efdf4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 shutdown_connections 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7efde80692b0 0x7efde806b760 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7efe000fbd20 0x7efe0010c7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 --2- 172.21.15.105:0/1144758942 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efe000fcfd0 0x7efe0010cce0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 >> 172.21.15.105:0/1144758942 conn(0x7efe000f7460 msgr2=0x7efe000f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 shutdown_connections 2024-03-24T11:15:30.251 INFO:teuthology.orchestra.run.smithi105.stderr:2024-03-24T11:15:30.250+0000 7efe064ab700 1 -- 172.21.15.105:0/1144758942 wait complete. 2024-03-24T11:15:30.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:30 smithi105 ceph-mon[36437]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:30.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:30 smithi100 ceph-mon[28941]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:30.863 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:15:30.863 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-03-24T11:15:30.863 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-03-24T11:15:30.908 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-03-24T11:15:30.908 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-03-24T11:15:30.908 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph mgr dump --format=json 2024-03-24T11:15:31.162 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:31.516 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:31 smithi100 ceph-mon[28941]: from='client.? 172.21.15.105:0/1144758942' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:15:31.516 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:31 smithi100 ceph-mon[28941]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:15:31.516 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:31 smithi100 ceph-mon[28941]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:15:31.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:31 smithi105 ceph-mon[36437]: from='client.? 172.21.15.105:0/1144758942' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:15:31.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:31 smithi105 ceph-mon[36437]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-03-24T11:15:31.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:31 smithi105 ceph-mon[36437]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-03-24T11:15:32.515 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:32 smithi100 ceph-mon[28941]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:32.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:32 smithi105 ceph-mon[36437]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:32.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 -- 172.21.15.100:0/1713465848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14180fb4d0 msgr2=0x7f14180fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:32.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 --2- 172.21.15.100:0/1713465848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14180fb4d0 0x7f14180fb940 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f140c004940 tx=0x7f140c009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:32.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 -- 172.21.15.100:0/1713465848 shutdown_connections 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 --2- 172.21.15.100:0/1713465848 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f14180fb4d0 0x7f14180fb940 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 --2- 172.21.15.100:0/1713465848 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1418102f50 0x7f1418103320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 -- 172.21.15.100:0/1713465848 >> 172.21.15.100:0/1713465848 conn(0x7f14180714c0 msgr2=0x7f14180718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 -- 172.21.15.100:0/1713465848 shutdown_connections 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.886+0000 7f141e0d7700 1 -- 172.21.15.100:0/1713465848 wait complete. 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f141e0d7700 1 Processor -- start 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f141e0d7700 1 -- start start 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f141e0d7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14180fb4d0 0x7f14181139a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f141e0d7700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 0x7f1418113ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f141e0d7700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1418114530 con 0x7f1418102f50 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f141e0d7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f141810db50 con 0x7f14180fb4d0 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f1416ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 0x7f1418113ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f1416ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 0x7f1418113ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:38390/0 (socket says 172.21.15.100:38390) 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f1416ffd700 1 -- 172.21.15.100:0/3467313994 learned_addr learned my addr 172.21.15.100:0/3467313994 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:32.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f14177fe700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14180fb4d0 0x7f14181139a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f1416ffd700 1 -- 172.21.15.100:0/3467313994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14180fb4d0 msgr2=0x7f14181139a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f1416ffd700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14180fb4d0 0x7f14181139a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.887+0000 7f1416ffd700 1 -- 172.21.15.100:0/3467313994 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f140c005040 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.888+0000 7f1416ffd700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 0x7f1418113ee0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b3b0 tx=0x7f140c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.888+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f140c00e040 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.888+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f140c00bbc0 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.888+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f141810ddd0 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.888+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f140c01c3f0 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.888+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f141810e310 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.889+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1418105330 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.889+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f140c00bd30 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.889+0000 7f13fffff700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f14000692a0 0x7f140006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.890+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f140c0880e0 con 0x7f1418102f50 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.890+0000 7f14177fe700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f14000692a0 0x7f140006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:32.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.890+0000 7f14177fe700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f14000692a0 0x7f140006b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1418065bf0 tx=0x7f1408009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:32.901 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:32.894+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f140c0546d0 con 0x7f1418102f50 2024-03-24T11:15:33.133 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.132+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f141810eff0 con 0x7f1418102f50 2024-03-24T11:15:33.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.135+0000 7f13fffff700 1 -- 172.21.15.100:0/3467313994 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+167176 (secure 0 0 0) 0x7f140c057cf0 con 0x7f1418102f50 2024-03-24T11:15:33.139 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:15:33.140 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.137+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f14000692a0 msgr2=0x7f140006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f14000692a0 0x7f140006b750 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7f1418065bf0 tx=0x7f1408009040 comp rx=0 tx=0).stop 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 msgr2=0x7f1418113ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 0x7f1418113ee0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f140c00b3b0 tx=0x7f140c009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 shutdown_connections 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f14000692a0 0x7f140006b750 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f14180fb4d0 0x7f14181139a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 --2- 172.21.15.100:0/3467313994 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1418102f50 0x7f1418113ee0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 >> 172.21.15.100:0/3467313994 conn(0x7f14180714c0 msgr2=0x7f14180fa720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 shutdown_connections 2024-03-24T11:15:33.141 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:33.138+0000 7f141e0d7700 1 -- 172.21.15.100:0/3467313994 wait complete. 2024-03-24T11:15:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:33 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3467313994' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:15:33.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:33 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3467313994' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-03-24T11:15:33.942 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":19,"active_gid":14210,"active_name":"smithi100.smxgyi","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6800","nonce":2223271137},{"type":"v1","addr":"172.21.15.100:6801","nonce":2223271137}]},"active_addr":"172.21.15.100:6801/2223271137","active_change":"2024-03-24T11:11:53.102165+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14230,"name":"smithi105.nmspmy","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":true,"error_string":"","module_options":{"ceph_event_retention_days":{"name":"ceph_event_retention_days","type":"int","level":"advanced","flags":0,"default_value":"7","min":"","max":"","enum_allowed":[],"desc":"Days to hold ceph event information within local cache","long_desc":"","tags":[],"see_also":[]},"config_check_secs":{"name":"config_check_secs","type":"int","level":"advanced","flags":0,"default_value":"10","min":"10","max":"","enum_allowed":[],"desc":"interval (secs) to check for cluster configuration changes","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.100:8443/","prometheus":"http://172.21.15.100:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":2085217808}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":3664315588}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":884193451}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.100:0","nonce":588047822}]}]} 2024-03-24T11:15:33.946 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-03-24T11:15:33.947 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-03-24T11:15:33.947 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:15:34.199 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:34.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:34 smithi105 ceph-mon[36437]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:34.563 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:34 smithi100 ceph-mon[28941]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:36.031 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 -- 172.21.15.100:0/3236908855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 msgr2=0x7fe6480fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:36.031 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 --2- 172.21.15.100:0/3236908855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe6480fce10 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7fe638004940 tx=0x7fe638009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 -- 172.21.15.100:0/3236908855 shutdown_connections 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 --2- 172.21.15.100:0/3236908855 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe6480fce10 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 --2- 172.21.15.100:0/3236908855 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe648102e00 0x7fe6481031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 -- 172.21.15.100:0/3236908855 >> 172.21.15.100:0/3236908855 conn(0x7fe6480f89b0 msgr2=0x7fe6480fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 -- 172.21.15.100:0/3236908855 shutdown_connections 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 -- 172.21.15.100:0/3236908855 wait complete. 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 Processor -- start 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.029+0000 7fe64f962700 1 -- start start 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64f962700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe648102e00 0x7fe6480702c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64f962700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe648070800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64f962700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6480722a0 con 0x7fe6481037a0 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64f962700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe648070d40 con 0x7fe648102e00 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe648070800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:36.032 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe648070800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:38400/0 (socket says 172.21.15.100:38400) 2024-03-24T11:15:36.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 -- 172.21.15.100:0/2739780701 learned_addr learned my addr 172.21.15.100:0/2739780701 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:36.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64d6fe700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe648102e00 0x7fe6480702c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:36.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 -- 172.21.15.100:0/2739780701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe648102e00 msgr2=0x7fe6480702c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:36.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe648102e00 0x7fe6480702c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:36.033 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 -- 172.21.15.100:0/2739780701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe638005040 con 0x7fe6481037a0 2024-03-24T11:15:36.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.030+0000 7fe64cefd700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe648070800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe638009180 tx=0x7fe638009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:36.034 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.031+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63800e040 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.031+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe648070fc0 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.031+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe638013420 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.031+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe63801c610 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.031+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64811a1d0 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.032+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe648100110 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.032+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe638013a00 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.033+0000 7fe63dffb700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe6340692a0 0x7fe63406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.033+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe638087f50 con 0x7fe6481037a0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.033+0000 7fe64d6fe700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe6340692a0 0x7fe63406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:36.035 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.033+0000 7fe64d6fe700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe6340692a0 0x7fe63406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe640007580 tx=0x7fe640009780 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:36.038 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.038+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe6380ba190 con 0x7fe6481037a0 2024-03-24T11:15:36.224 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.223+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe6480fe550 con 0x7fe6481037a0 2024-03-24T11:15:36.225 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.223+0000 7fe63dffb700 1 -- 172.21.15.100:0/2739780701 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11135 (secure 0 0 0) 0x7fe6380545c0 con 0x7fe6481037a0 2024-03-24T11:15:36.225 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:15:36.225 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":41,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","created":"2024-03-24T11:09:42.710984+0000","modified":"2024-03-24T11:15:18.868228+0000","last_up_change":"2024-03-24T11:15:17.863076+0000","last_in_change":"2024-03-24T11:14:58.878641+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:14:05.424502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cc6e0f08-af11-46b2-923f-17e549042b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6803","nonce":3447712544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6805","nonce":3447712544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6809","nonce":3447712544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6807","nonce":3447712544}]},"public_addr":"172.21.15.100:6803/3447712544","cluster_addr":"172.21.15.100:6805/3447712544","heartbeat_back_addr":"172.21.15.100:6809/3447712544","heartbeat_front_addr":"172.21.15.100:6807/3447712544","state":["exists","up"]},{"osd":1,"uuid":"3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6811","nonce":1490052239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6813","nonce":1490052239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6817","nonce":1490052239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6815","nonce":1490052239}]},"public_addr":"172.21.15.100:6811/1490052239","cluster_addr":"172.21.15.100:6813/1490052239","heartbeat_back_addr":"172.21.15.100:6817/1490052239","heartbeat_front_addr":"172.21.15.100:6815/1490052239","state":["exists","up"]},{"osd":2,"uuid":"44db3f0c-2877-4026-93f3-febdb7f46d67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6819","nonce":2671131686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6821","nonce":2671131686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6825","nonce":2671131686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6823","nonce":2671131686}]},"public_addr":"172.21.15.100:6819/2671131686","cluster_addr":"172.21.15.100:6821/2671131686","heartbeat_back_addr":"172.21.15.100:6825/2671131686","heartbeat_front_addr":"172.21.15.100:6823/2671131686","state":["exists","up"]},{"osd":3,"uuid":"1f6ededc-bddf-4308-a416-80ae19ad3fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6801","nonce":3221138489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6803","nonce":3221138489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6807","nonce":3221138489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6805","nonce":3221138489}]},"public_addr":"172.21.15.105:6801/3221138489","cluster_addr":"172.21.15.105:6803/3221138489","heartbeat_back_addr":"172.21.15.105:6807/3221138489","heartbeat_front_addr":"172.21.15.105:6805/3221138489","state":["exists","up"]},{"osd":4,"uuid":"c59e7571-57a0-406f-bdbb-614deeaaa56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6809","nonce":3756920843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6811","nonce":3756920843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6815","nonce":3756920843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6813","nonce":3756920843}]},"public_addr":"172.21.15.105:6809/3756920843","cluster_addr":"172.21.15.105:6811/3756920843","heartbeat_back_addr":"172.21.15.105:6815/3756920843","heartbeat_front_addr":"172.21.15.105:6813/3756920843","state":["exists","up"]},{"osd":5,"uuid":"15b5ed12-6a0d-4bfe-9ffd-5224e21353ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6817","nonce":256673904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6819","nonce":256673904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6823","nonce":256673904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6821","nonce":256673904}]},"public_addr":"172.21.15.105:6817/256673904","cluster_addr":"172.21.15.105:6819/256673904","heartbeat_back_addr":"172.21.15.105:6823/256673904","heartbeat_front_addr":"172.21.15.105:6821/256673904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:03.013039+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:32.779377+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:01.889286+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:25.258963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:50.202419+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:15:16.423165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/2476598760":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/3850505170":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6801/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/374004208":"2024-03-25T11:10:47.648919+0000","172.21.15.100:6801/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/3084685173":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/1681068182":"2024-03-25T11:10:16.917673+0000","172.21.15.100:0/3561410096":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2555446076":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6801/1648043756":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6800/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2493584253":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/4249885134":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1648043756":"2024-03-25T11:10:16.917673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe6340692a0 msgr2=0x7fe63406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe6340692a0 0x7fe63406b750 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fe640007580 tx=0x7fe640009780 comp rx=0 tx=0).stop 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 msgr2=0x7fe648070800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe648070800 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fe638009180 tx=0x7fe638009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 shutdown_connections 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe6340692a0 0x7fe63406b750 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe648102e00 0x7fe6480702c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 --2- 172.21.15.100:0/2739780701 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe6481037a0 0x7fe648070800 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:36.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.225+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 >> 172.21.15.100:0/2739780701 conn(0x7fe6480f89b0 msgr2=0x7fe6480fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:36.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.226+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 shutdown_connections 2024-03-24T11:15:36.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:36.226+0000 7fe64f962700 1 -- 172.21.15.100:0/2739780701 wait complete. 2024-03-24T11:15:36.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:36 smithi100 ceph-mon[28941]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:36.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:36 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2739780701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:15:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:36 smithi105 ceph-mon[36437]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:36 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2739780701' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:15:37.025 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-03-24T11:15:37.025 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd dump --format=json 2024-03-24T11:15:37.280 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:38.541 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:38 smithi100 ceph-mon[28941]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:38.541 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:38 smithi105 ceph-mon[36437]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 -- 172.21.15.100:0/1011841504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0701037a0 msgr2=0x7fb0700fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 --2- 172.21.15.100:0/1011841504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0701037a0 0x7fb0700fce10 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb05c004940 tx=0x7fb05c009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 -- 172.21.15.100:0/1011841504 shutdown_connections 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 --2- 172.21.15.100:0/1011841504 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0701037a0 0x7fb0700fce10 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 --2- 172.21.15.100:0/1011841504 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb070102e00 0x7fb0701031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 -- 172.21.15.100:0/1011841504 >> 172.21.15.100:0/1011841504 conn(0x7fb0700f89b0 msgr2=0x7fb0700fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 -- 172.21.15.100:0/1011841504 shutdown_connections 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 -- 172.21.15.100:0/1011841504 wait complete. 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 Processor -- start 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.084+0000 7fb076ff1700 1 -- start start 2024-03-24T11:15:39.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb076ff1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 0x7fb070113990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb076ff1700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0701037a0 0x7fb070113ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb076ff1700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0701145b0 con 0x7fb070102e00 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb076ff1700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb07010db40 con 0x7fb0701037a0 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 0x7fb070113990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 0x7fb070113990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43106/0 (socket says 172.21.15.100:43106) 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 -- 172.21.15.100:0/132612475 learned_addr learned my addr 172.21.15.100:0/132612475 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 -- 172.21.15.100:0/132612475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0701037a0 msgr2=0x7fb070113ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0701037a0 0x7fb070113ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 -- 172.21.15.100:0/132612475 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb05c005040 con 0x7fb070102e00 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb074d8d700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 0x7fb070113990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb06400b790 tx=0x7fb064009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06400f770 con 0x7fb070102e00 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb06400fda0 con 0x7fb070102e00 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb07010de20 con 0x7fb070102e00 2024-03-24T11:15:39.087 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb06400d310 con 0x7fb070102e00 2024-03-24T11:15:39.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.085+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb07010e3e0 con 0x7fb070102e00 2024-03-24T11:15:39.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.086+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb070100110 con 0x7fb070102e00 2024-03-24T11:15:39.088 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.087+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb064010420 con 0x7fb070102e00 2024-03-24T11:15:39.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.087+0000 7fb06d7fa700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0600692a0 0x7fb06006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:39.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.087+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb064086e60 con 0x7fb070102e00 2024-03-24T11:15:39.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.087+0000 7fb06ffff700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0600692a0 0x7fb06006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:39.089 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.088+0000 7fb06ffff700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0600692a0 0x7fb06006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb05c004f90 tx=0x7fb05c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:39.093 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.093+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb0640534d0 con 0x7fb070102e00 2024-03-24T11:15:39.280 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.280+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb0700fe550 con 0x7fb070102e00 2024-03-24T11:15:39.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.280+0000 7fb06d7fa700 1 -- 172.21.15.100:0/132612475 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11135 (secure 0 0 0) 0x7fb064056af0 con 0x7fb070102e00 2024-03-24T11:15:39.281 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:15:39.281 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":41,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","created":"2024-03-24T11:09:42.710984+0000","modified":"2024-03-24T11:15:18.868228+0000","last_up_change":"2024-03-24T11:15:17.863076+0000","last_in_change":"2024-03-24T11:14:58.878641+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:14:05.424502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"cc6e0f08-af11-46b2-923f-17e549042b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6803","nonce":3447712544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6805","nonce":3447712544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6809","nonce":3447712544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6807","nonce":3447712544}]},"public_addr":"172.21.15.100:6803/3447712544","cluster_addr":"172.21.15.100:6805/3447712544","heartbeat_back_addr":"172.21.15.100:6809/3447712544","heartbeat_front_addr":"172.21.15.100:6807/3447712544","state":["exists","up"]},{"osd":1,"uuid":"3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6811","nonce":1490052239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6813","nonce":1490052239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6817","nonce":1490052239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6815","nonce":1490052239}]},"public_addr":"172.21.15.100:6811/1490052239","cluster_addr":"172.21.15.100:6813/1490052239","heartbeat_back_addr":"172.21.15.100:6817/1490052239","heartbeat_front_addr":"172.21.15.100:6815/1490052239","state":["exists","up"]},{"osd":2,"uuid":"44db3f0c-2877-4026-93f3-febdb7f46d67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6819","nonce":2671131686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6821","nonce":2671131686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6825","nonce":2671131686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6823","nonce":2671131686}]},"public_addr":"172.21.15.100:6819/2671131686","cluster_addr":"172.21.15.100:6821/2671131686","heartbeat_back_addr":"172.21.15.100:6825/2671131686","heartbeat_front_addr":"172.21.15.100:6823/2671131686","state":["exists","up"]},{"osd":3,"uuid":"1f6ededc-bddf-4308-a416-80ae19ad3fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6801","nonce":3221138489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6803","nonce":3221138489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6807","nonce":3221138489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6805","nonce":3221138489}]},"public_addr":"172.21.15.105:6801/3221138489","cluster_addr":"172.21.15.105:6803/3221138489","heartbeat_back_addr":"172.21.15.105:6807/3221138489","heartbeat_front_addr":"172.21.15.105:6805/3221138489","state":["exists","up"]},{"osd":4,"uuid":"c59e7571-57a0-406f-bdbb-614deeaaa56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6809","nonce":3756920843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6811","nonce":3756920843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6815","nonce":3756920843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6813","nonce":3756920843}]},"public_addr":"172.21.15.105:6809/3756920843","cluster_addr":"172.21.15.105:6811/3756920843","heartbeat_back_addr":"172.21.15.105:6815/3756920843","heartbeat_front_addr":"172.21.15.105:6813/3756920843","state":["exists","up"]},{"osd":5,"uuid":"15b5ed12-6a0d-4bfe-9ffd-5224e21353ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6817","nonce":256673904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6819","nonce":256673904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6823","nonce":256673904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6821","nonce":256673904}]},"public_addr":"172.21.15.105:6817/256673904","cluster_addr":"172.21.15.105:6819/256673904","heartbeat_back_addr":"172.21.15.105:6823/256673904","heartbeat_front_addr":"172.21.15.105:6821/256673904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:03.013039+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:32.779377+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:01.889286+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:25.258963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:50.202419+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:15:16.423165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/2476598760":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/3850505170":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6801/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/374004208":"2024-03-25T11:10:47.648919+0000","172.21.15.100:6801/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/3084685173":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/1681068182":"2024-03-25T11:10:16.917673+0000","172.21.15.100:0/3561410096":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2555446076":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6801/1648043756":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6800/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2493584253":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/4249885134":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1648043756":"2024-03-25T11:10:16.917673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:15:39.283 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.282+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0600692a0 msgr2=0x7fb06006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:39.283 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0600692a0 0x7fb06006b750 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb05c004f90 tx=0x7fb05c009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 msgr2=0x7fb070113990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 0x7fb070113990 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fb06400b790 tx=0x7fb064009250 comp rx=0 tx=0).stop 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 shutdown_connections 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0600692a0 0x7fb06006b750 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb070102e00 0x7fb070113990 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 --2- 172.21.15.100:0/132612475 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0701037a0 0x7fb070113ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:39.284 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 >> 172.21.15.100:0/132612475 conn(0x7fb0700f89b0 msgr2=0x7fb0700fa770 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:39.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 shutdown_connections 2024-03-24T11:15:39.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:39.283+0000 7fb076ff1700 1 -- 172.21.15.100:0/132612475 wait complete. 2024-03-24T11:15:40.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph tell osd.0 flush_pg_stats 2024-03-24T11:15:40.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph tell osd.1 flush_pg_stats 2024-03-24T11:15:40.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph tell osd.2 flush_pg_stats 2024-03-24T11:15:40.016 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph tell osd.3 flush_pg_stats 2024-03-24T11:15:40.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph tell osd.4 flush_pg_stats 2024-03-24T11:15:40.017 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph tell osd.5 flush_pg_stats 2024-03-24T11:15:40.263 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:40 smithi100 ceph-mon[28941]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:40.264 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:40 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/132612475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:15:40.340 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:40.348 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:40 smithi105 ceph-mon[36437]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:40 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/132612475' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:15:41.042 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:41.042 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:42.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:42 smithi100 ceph-mon[28941]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:42 smithi105 ceph-mon[36437]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:44.510 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:44 smithi100 ceph-mon[28941]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:44 smithi105 ceph-mon[36437]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:44.857 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:44.857 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:46 smithi105 ceph-mon[36437]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:46.596 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:46 smithi100 ceph-mon[28941]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:47.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 -- 172.21.15.100:0/2625382272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 msgr2=0x7feff80fb3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 --2- 172.21.15.100:0/2625382272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff80fb3b0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fefe4004940 tx=0x7fefe4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 -- 172.21.15.100:0/2625382272 shutdown_connections 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 --2- 172.21.15.100:0/2625382272 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff80fb3b0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 --2- 172.21.15.100:0/2625382272 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff8100c40 0x7feff8101010 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 -- 172.21.15.100:0/2625382272 >> 172.21.15.100:0/2625382272 conn(0x7feff8072310 msgr2=0x7feff8072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 -- 172.21.15.100:0/2625382272 shutdown_connections 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 -- 172.21.15.100:0/2625382272 wait complete. 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 Processor -- start 2024-03-24T11:15:47.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.985+0000 7feffee96700 1 -- start start 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feffee96700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff8100c40 0x7feff8119f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feffee96700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff811a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feffee96700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff811ab00 con 0x7feff81015e0 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feffee96700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff81140e0 con 0x7feff8100c40 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff811a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff811a4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56704/0 (socket says 172.21.15.100:56704) 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 -- 172.21.15.100:0/2594224291 learned_addr learned my addr 172.21.15.100:0/2594224291 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feffcc32700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff8100c40 0x7feff8119f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 -- 172.21.15.100:0/2594224291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff8100c40 msgr2=0x7feff8119f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff8100c40 0x7feff8119f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 -- 172.21.15.100:0/2594224291 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe4005040 con 0x7feff81015e0 2024-03-24T11:15:47.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff7fff700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff811a4b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009100 tx=0x7fefe4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:47.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe400e040 con 0x7feff81015e0 2024-03-24T11:15:47.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff8114360 con 0x7feff81015e0 2024-03-24T11:15:47.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.986+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fefe4009af0 con 0x7feff81015e0 2024-03-24T11:15:47.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.987+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fefe401c3f0 con 0x7feff81015e0 2024-03-24T11:15:47.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.987+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff8114920 con 0x7feff81015e0 2024-03-24T11:15:47.989 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.987+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feff8114e00 con 0x7feff81015e0 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.988+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fefe400baf0 con 0x7feff81015e0 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.988+0000 7feff57fa700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fefe80692a0 0x7fefe806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.988+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fefe40880d0 con 0x7feff81015e0 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.988+0000 7feffcc32700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fefe80692a0 0x7fefe806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.988+0000 7feff57fa700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] conn(0x7fefe806ef70 0x7fefe8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.988+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 --> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fefe8071a30 con 0x7fefe806ef70 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.989+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fefe408a6c0 con 0x7feff81015e0 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.989+0000 7feffd433700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] conn(0x7fefe806ef70 0x7fefe8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:47.990 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.989+0000 7feffcc32700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fefe80692a0 0x7fefe806b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fefec004eb0 tx=0x7fefec00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:47.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.989+0000 7feffd433700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] conn(0x7fefe806ef70 0x7fefe8071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:47.991 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:47.990+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== osd.1 v2:172.21.15.100:6810/1490052239 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fefe8071a30 con 0x7fefe806ef70 2024-03-24T11:15:48.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.004+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 --> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feff805e6c0 con 0x7fefe806ef70 2024-03-24T11:15:48.005 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feff57fa700 1 -- 172.21.15.100:0/2594224291 <== osd.1 v2:172.21.15.100:6810/1490052239 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7feff805e6c0 con 0x7fefe806ef70 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] conn(0x7fefe806ef70 msgr2=0x7fefe8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] conn(0x7fefe806ef70 0x7fefe8071380 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fefe80692a0 msgr2=0x7fefe806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fefe80692a0 0x7fefe806b750 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fefec004eb0 tx=0x7fefec00b040 comp rx=0 tx=0).stop 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 msgr2=0x7feff811a4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff811a4b0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7fefe4009100 tx=0x7fefe4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 shutdown_connections 2024-03-24T11:15:48.006 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fefe80692a0 0x7fefe806b750 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feff8100c40 0x7feff8119f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:6810/1490052239,v1:172.21.15.100:6811/1490052239] conn(0x7fefe806ef70 0x7fefe8071380 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 --2- 172.21.15.100:0/2594224291 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feff81015e0 0x7feff811a4b0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 >> 172.21.15.100:0/2594224291 conn(0x7feff8072310 msgr2=0x7feff80fe010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:48.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 shutdown_connections 2024-03-24T11:15:48.007 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.005+0000 7feffee96700 1 -- 172.21.15.100:0/2594224291 wait complete. 2024-03-24T11:15:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:48 smithi105 ceph-mon[36437]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:48.562 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:48 smithi100 ceph-mon[28941]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:48.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 -- 172.21.15.100:0/240790098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20101d70 msgr2=0x7f4b20102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 --2- 172.21.15.100:0/240790098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20101d70 0x7f4b20102140 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7f4b18009ce0 tx=0x7f4b18004d10 comp rx=0 tx=0).stop 2024-03-24T11:15:48.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 -- 172.21.15.100:0/240790098 shutdown_connections 2024-03-24T11:15:48.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 --2- 172.21.15.100:0/240790098 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20102710 0x7f4b200fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 --2- 172.21.15.100:0/240790098 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20101d70 0x7f4b20102140 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.870 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 -- 172.21.15.100:0/240790098 >> 172.21.15.100:0/240790098 conn(0x7f4b20072bc0 msgr2=0x7f4b20072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:48.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 -- 172.21.15.100:0/240790098 shutdown_connections 2024-03-24T11:15:48.871 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.869+0000 7f4b256c4700 1 -- 172.21.15.100:0/240790098 wait complete. 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b256c4700 1 Processor -- start 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b256c4700 1 -- start start 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b256c4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20101d70 0x7f4b20113c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b256c4700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 0x7f4b20114190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b256c4700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b201147e0 con 0x7f4b20102710 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b256c4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2010dc60 con 0x7f4b20101d70 2024-03-24T11:15:48.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.870+0000 7f4b1e7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 0x7f4b20114190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1e7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 0x7f4b20114190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56742/0 (socket says 172.21.15.100:56742) 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1e7fc700 1 -- 172.21.15.100:0/1723649874 learned_addr learned my addr 172.21.15.100:0/1723649874 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1effd700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20101d70 0x7f4b20113c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1e7fc700 1 -- 172.21.15.100:0/1723649874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20101d70 msgr2=0x7f4b20113c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1e7fc700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20101d70 0x7f4b20113c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1e7fc700 1 -- 172.21.15.100:0/1723649874 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b18009940 con 0x7f4b20102710 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1effd700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20101d70 0x7f4b20113c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b1e7fc700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 0x7f4b20114190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4b08002a80 tx=0x7f4b08004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b08007cc0 con 0x7f4b20102710 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b2010df40 con 0x7f4b20102710 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4b0800f040 con 0x7f4b20102710 2024-03-24T11:15:48.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4b0800daf0 con 0x7f4b20102710 2024-03-24T11:15:48.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.871+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b2010e500 con 0x7f4b20102710 2024-03-24T11:15:48.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.872+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b2010e9e0 con 0x7f4b20102710 2024-03-24T11:15:48.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4b0800dc50 con 0x7f4b20102710 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b177fe700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4b0c0692a0 0x7f4b0c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4b08086a10 con 0x7f4b20102710 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b1effd700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4b0c0692a0 0x7f4b0c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b177fe700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] conn(0x7f4b0c06ef70 0x7f4b0c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 --> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4b0c071a30 con 0x7f4b0c06ef70 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.873+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4b08086df0 con 0x7f4b20102710 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.874+0000 7f4b1f7fe700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] conn(0x7f4b0c06ef70 0x7f4b0c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.874+0000 7f4b1effd700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4b0c0692a0 0x7f4b0c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b180048c0 tx=0x7f4b18004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.874+0000 7f4b1f7fe700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] conn(0x7f4b0c06ef70 0x7f4b0c071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:48.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.874+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== osd.0 v2:172.21.15.100:6802/3447712544 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4b0c071a30 con 0x7f4b0c06ef70 2024-03-24T11:15:48.890 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.889+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 --> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b20105fe0 con 0x7f4b0c06ef70 2024-03-24T11:15:48.890 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b177fe700 1 -- 172.21.15.100:0/1723649874 <== osd.0 v2:172.21.15.100:6802/3447712544 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f4b20105fe0 con 0x7f4b0c06ef70 2024-03-24T11:15:48.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] conn(0x7f4b0c06ef70 msgr2=0x7f4b0c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] conn(0x7f4b0c06ef70 0x7f4b0c071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4b0c0692a0 msgr2=0x7f4b0c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4b0c0692a0 0x7f4b0c06b750 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7f4b180048c0 tx=0x7f4b18004d10 comp rx=0 tx=0).stop 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 msgr2=0x7f4b20114190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 0x7f4b20114190 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7f4b08002a80 tx=0x7f4b08004630 comp rx=0 tx=0).stop 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 shutdown_connections 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4b0c0692a0 0x7f4b0c06b750 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:6802/3447712544,v1:172.21.15.100:6803/3447712544] conn(0x7f4b0c06ef70 0x7f4b0c071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b20101d70 0x7f4b20113c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 --2- 172.21.15.100:0/1723649874 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b20102710 0x7f4b20114190 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 >> 172.21.15.100:0/1723649874 conn(0x7f4b20072bc0 msgr2=0x7f4b20065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 shutdown_connections 2024-03-24T11:15:48.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.890+0000 7f4b256c4700 1 -- 172.21.15.100:0/1723649874 wait complete. 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 -- 172.21.15.100:0/3216317249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 msgr2=0x7f4b001020e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 --2- 172.21.15.100:0/3216317249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b001020e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f4af0004940 tx=0x7f4af0009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 -- 172.21.15.100:0/3216317249 shutdown_connections 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 --2- 172.21.15.100:0/3216317249 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 0x7f4b000fbd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 --2- 172.21.15.100:0/3216317249 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b001020e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 -- 172.21.15.100:0/3216317249 >> 172.21.15.100:0/3216317249 conn(0x7f4b000f7880 msgr2=0x7f4b000f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 -- 172.21.15.100:0/3216317249 shutdown_connections 2024-03-24T11:15:49.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.764+0000 7f4b06523700 1 -- 172.21.15.100:0/3216317249 wait complete. 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4b06523700 1 Processor -- start 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4b06523700 1 -- start start 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4b06523700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b0010ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4b06523700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 0x7f4b0010f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4b06523700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0010f9b0 con 0x7f4b00101d10 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4b06523700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b0010faf0 con 0x7f4b001026b0 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b0010ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b0010ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56716/0 (socket says 172.21.15.100:56716) 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 -- 172.21.15.100:0/1981806693 learned_addr learned my addr 172.21.15.100:0/1981806693 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4aff7fe700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 0x7f4b0010f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 -- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 msgr2=0x7f4b0010f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 0x7f4b0010f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 -- 172.21.15.100:0/1981806693 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4af0005040 con 0x7f4b00101d10 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4aff7fe700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 0x7f4b0010f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T11:15:49.095 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.765+0000 7f4afffff700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b0010ee20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4af000b3b0 tx=0x7f4af0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.766+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af000e040 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.766+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4af000bbc0 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.766+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b00117e90 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.766+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4af001c3f0 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.766+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b00118370 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.766+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f4b000fd460 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.767+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4af000bd30 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4afcff9700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4ae80692a0 0x7f4ae806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4af0088a60 con 0x7f4b00101d10 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4aff7fe700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4ae80692a0 0x7f4ae806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4afcff9700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] conn(0x7f4ae806ef70 0x7f4ae8071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 --> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f4ae8071a30 con 0x7f4ae806ef70 2024-03-24T11:15:49.096 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f4af0088e40 con 0x7f4b00101d10 2024-03-24T11:15:49.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4aff7fe700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4ae80692a0 0x7f4ae806b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009fd0 tx=0x7f4af400b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:49.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4b04ac0700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] conn(0x7f4ae806ef70 0x7f4ae8071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:49.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.768+0000 7f4b04ac0700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] conn(0x7f4ae806ef70 0x7f4ae8071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:49.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.769+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== osd.4 v2:172.21.15.105:6808/3756920843 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f4ae8071a30 con 0x7f4ae806ef70 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.784+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 --> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f4b0005f5f0 con 0x7f4ae806ef70 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.784+0000 7f4afcff9700 1 -- 172.21.15.100:0/1981806693 <== osd.4 v2:172.21.15.105:6808/3756920843 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f4b0005f5f0 con 0x7f4ae806ef70 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] conn(0x7f4ae806ef70 msgr2=0x7f4ae8071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] conn(0x7f4ae806ef70 0x7f4ae8071380 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4ae80692a0 msgr2=0x7f4ae806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4ae80692a0 0x7f4ae806b750 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7f4af4009fd0 tx=0x7f4af400b040 comp rx=0 tx=0).stop 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 msgr2=0x7f4b0010ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b0010ee20 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7f4af000b3b0 tx=0x7f4af0009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 shutdown_connections 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:6808/3756920843,v1:172.21.15.105:6809/3756920843] conn(0x7f4ae806ef70 0x7f4ae8071380 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4ae80692a0 0x7f4ae806b750 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4b00101d10 0x7f4b0010ee20 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.098 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 --2- 172.21.15.100:0/1981806693 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4b001026b0 0x7f4b0010f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:49.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 >> 172.21.15.100:0/1981806693 conn(0x7f4b000f7880 msgr2=0x7f4b000f8900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:49.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 shutdown_connections 2024-03-24T11:15:49.099 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:48.785+0000 7f4b06523700 1 -- 172.21.15.100:0/1981806693 wait complete. 2024-03-24T11:15:49.767 INFO:teuthology.orchestra.run.smithi100.stdout:150323855374 2024-03-24T11:15:49.768 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.4 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 -- 172.21.15.100:0/3473854433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c102f70 msgr2=0x7f301c103340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 --2- 172.21.15.100:0/3473854433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c102f70 0x7f301c103340 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7f3010004940 tx=0x7f3010009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 -- 172.21.15.100:0/3473854433 shutdown_connections 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 --2- 172.21.15.100:0/3473854433 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f301c0fb3b0 0x7f301c0fb820 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 --2- 172.21.15.100:0/3473854433 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c102f70 0x7f301c103340 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 -- 172.21.15.100:0/3473854433 >> 172.21.15.100:0/3473854433 conn(0x7f301c072310 msgr2=0x7f301c072710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 -- 172.21.15.100:0/3473854433 shutdown_connections 2024-03-24T11:15:50.189 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 -- 172.21.15.100:0/3473854433 wait complete. 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 Processor -- start 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.187+0000 7f3022f00700 1 -- start start 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3022f00700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 0x7f301c1139f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3022f00700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f301c102f70 0x7f301c113f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3022f00700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c114570 con 0x7f301c0fb3b0 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3022f00700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f301c10db30 con 0x7f301c102f70 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 0x7f301c1139f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 0x7f301c1139f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56752/0 (socket says 172.21.15.100:56752) 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 -- 172.21.15.100:0/3279497357 learned_addr learned my addr 172.21.15.100:0/3279497357 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 -- 172.21.15.100:0/3279497357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f301c102f70 msgr2=0x7f301c113f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f301c102f70 0x7f301c113f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 -- 172.21.15.100:0/3279497357 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3010005040 con 0x7f301c0fb3b0 2024-03-24T11:15:50.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3020c9c700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 0x7f301c1139f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f301000b3b0 tx=0x7f3010009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:50.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f301000e040 con 0x7f301c0fb3b0 2024-03-24T11:15:50.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f301c10ddb0 con 0x7f301c0fb3b0 2024-03-24T11:15:50.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f301000bbc0 con 0x7f301c0fb3b0 2024-03-24T11:15:50.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f301001c3f0 con 0x7f301c0fb3b0 2024-03-24T11:15:50.191 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.188+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f301c10e2f0 con 0x7f301c0fb3b0 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.189+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f301c10e810 con 0x7f301c0fb3b0 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f301000bd30 con 0x7f301c0fb3b0 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f30197fa700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f300c0692a0 0x7f300c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3010088b60 con 0x7f301c0fb3b0 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f301bfff700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f300c0692a0 0x7f300c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f30197fa700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] conn(0x7f300c06ef70 0x7f300c071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:50.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 --> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f300c071a30 con 0x7f300c06ef70 2024-03-24T11:15:50.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.190+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f3010088f40 con 0x7f301c0fb3b0 2024-03-24T11:15:50.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.191+0000 7f302149d700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] conn(0x7f300c06ef70 0x7f300c071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:50.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.191+0000 7f301bfff700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f300c0692a0 0x7f300c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3008006fd0 tx=0x7f3008006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:50.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.191+0000 7f302149d700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] conn(0x7f300c06ef70 0x7f300c071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:50.193 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.191+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== osd.2 v2:172.21.15.100:6818/2671131686 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f300c071a30 con 0x7f300c06ef70 2024-03-24T11:15:50.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.206+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 --> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f301c10e9d0 con 0x7f300c06ef70 2024-03-24T11:15:50.206 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.206+0000 7f30197fa700 1 -- 172.21.15.100:0/3279497357 <== osd.2 v2:172.21.15.100:6818/2671131686 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f301c10e9d0 con 0x7f300c06ef70 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] conn(0x7f300c06ef70 msgr2=0x7f300c071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] conn(0x7f300c06ef70 0x7f300c071380 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f300c0692a0 msgr2=0x7f300c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f300c0692a0 0x7f300c06b750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f3008006fd0 tx=0x7f3008006db0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 msgr2=0x7f301c1139f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 0x7f301c1139f0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f301000b3b0 tx=0x7f3010009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 shutdown_connections 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6818/2671131686,v1:172.21.15.100:6819/2671131686] conn(0x7f300c06ef70 0x7f300c071380 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f300c0692a0 0x7f300c06b750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f301c0fb3b0 0x7f301c1139f0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 --2- 172.21.15.100:0/3279497357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f301c102f70 0x7f301c113f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:50.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 >> 172.21.15.100:0/3279497357 conn(0x7f301c072310 msgr2=0x7f301c0f9580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:50.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 shutdown_connections 2024-03-24T11:15:50.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:50.207+0000 7f3022f00700 1 -- 172.21.15.100:0/3279497357 wait complete. 2024-03-24T11:15:50.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:50 smithi100 ceph-mon[28941]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:50 smithi105 ceph-mon[36437]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:51.542 INFO:teuthology.orchestra.run.smithi100.stdout:81604378648 2024-03-24T11:15:51.542 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.2 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 -- 172.21.15.100:0/1990421107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 msgr2=0x7feae00fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 --2- 172.21.15.100:0/1990421107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae00fbd80 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7feadc004940 tx=0x7feadc009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 -- 172.21.15.100:0/1990421107 shutdown_connections 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 --2- 172.21.15.100:0/1990421107 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae00fbd80 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 --2- 172.21.15.100:0/1990421107 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae0101d70 0x7feae0102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 -- 172.21.15.100:0/1990421107 >> 172.21.15.100:0/1990421107 conn(0x7feae0072bc0 msgr2=0x7feae0072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:51.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.964+0000 7feae7e05700 1 -- 172.21.15.100:0/1990421107 shutdown_connections 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.965+0000 7feae7e05700 1 -- 172.21.15.100:0/1990421107 wait complete. 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.965+0000 7feae7e05700 1 Processor -- start 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.965+0000 7feae7e05700 1 -- start start 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.965+0000 7feae7e05700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae0101d70 0x7feae010eb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae7e05700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae010f070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae7e05700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae010f750 con 0x7feae0102710 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae7e05700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feae010f890 con 0x7feae0101d70 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae010f070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:51.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae010f070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56762/0 (socket says 172.21.15.100:56762) 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 -- 172.21.15.100:0/2893345656 learned_addr learned my addr 172.21.15.100:0/2893345656 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae5ba1700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae0101d70 0x7feae010eb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 -- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae0101d70 msgr2=0x7feae010eb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae0101d70 0x7feae010eb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 -- 172.21.15.100:0/2893345656 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feadc005040 con 0x7feae0102710 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae53a0700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae010f070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feadc009100 tx=0x7feadc009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feadc00e040 con 0x7feae0102710 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7feae7e05700 1 -- 172.21.15.100:0/2893345656 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feae0117ba0 con 0x7feae0102710 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7feadc009af0 con 0x7feae0102710 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.966+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feadc01c3f0 con 0x7feae0102710 2024-03-24T11:15:51.968 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.967+0000 7feae7e05700 1 -- 172.21.15.100:0/2893345656 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feae0118130 con 0x7feae0102710 2024-03-24T11:15:51.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.967+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7feae005e6c0 con 0x7feae0102710 2024-03-24T11:15:51.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.968+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7feadc00baf0 con 0x7feae0102710 2024-03-24T11:15:51.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.968+0000 7fead67fc700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7feacc0692a0 0x7feacc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.968+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feadc088170 con 0x7feae0102710 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.968+0000 7feae5ba1700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7feacc0692a0 0x7feacc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.969+0000 7fead67fc700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] conn(0x7feacc06ef70 0x7feacc071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.969+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 --> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7feacc071a30 con 0x7feacc06ef70 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.969+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7feadc08a720 con 0x7feae0102710 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.969+0000 7feae5ba1700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7feacc0692a0 0x7feacc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fead0009b60 tx=0x7fead00091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.969+0000 7feae63a2700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] conn(0x7feacc06ef70 0x7feacc071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.969+0000 7feae63a2700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] conn(0x7feacc06ef70 0x7feacc071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:51.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.970+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== osd.5 v2:172.21.15.105:6816/256673904 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7feacc071a30 con 0x7feacc06ef70 2024-03-24T11:15:51.985 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.985+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 --> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7feae005eaf0 con 0x7feacc06ef70 2024-03-24T11:15:51.986 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7fead67fc700 1 -- 172.21.15.100:0/2893345656 <== osd.5 v2:172.21.15.105:6816/256673904 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7feae005eaf0 con 0x7feacc06ef70 2024-03-24T11:15:51.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] conn(0x7feacc06ef70 msgr2=0x7feacc071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:51.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] conn(0x7feacc06ef70 0x7feacc071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7feacc0692a0 msgr2=0x7feacc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:51.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7feacc0692a0 0x7feacc06b750 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fead0009b60 tx=0x7fead00091a0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 msgr2=0x7feae010f070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:51.987 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae010f070 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7feadc009100 tx=0x7feadc009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 shutdown_connections 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7feacc0692a0 0x7feacc06b750 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:6816/256673904,v1:172.21.15.105:6817/256673904] conn(0x7feacc06ef70 0x7feacc071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feae0101d70 0x7feae010eb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 --2- 172.21.15.100:0/2893345656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7feae0102710 0x7feae010f070 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.986+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 >> 172.21.15.100:0/2893345656 conn(0x7feae0072bc0 msgr2=0x7feae00faf20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.987+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 shutdown_connections 2024-03-24T11:15:51.988 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:51.987+0000 7feacbfff700 1 -- 172.21.15.100:0/2893345656 wait complete. 2024-03-24T11:15:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:52 smithi105 ceph-mon[36437]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:52.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:52 smithi100 ceph-mon[28941]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:52.875 INFO:teuthology.orchestra.run.smithi100.stdout:171798691850 2024-03-24T11:15:52.875 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.5 2024-03-24T11:15:53.384 INFO:teuthology.orchestra.run.smithi100.stdout:38654705699 2024-03-24T11:15:53.385 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.0 2024-03-24T11:15:53.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:53.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:15:53.711 INFO:teuthology.orchestra.run.smithi100.stdout:60129542174 2024-03-24T11:15:53.712 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.1 2024-03-24T11:15:53.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 -- 172.21.15.100:0/1887180929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 msgr2=0x7f52e80fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:53.907 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 --2- 172.21.15.100:0/1887180929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e80fbd80 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7f52d8004940 tx=0x7f52d8009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 -- 172.21.15.100:0/1887180929 shutdown_connections 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 --2- 172.21.15.100:0/1887180929 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e80fbd80 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 --2- 172.21.15.100:0/1887180929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 0x7f52e8102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 -- 172.21.15.100:0/1887180929 >> 172.21.15.100:0/1887180929 conn(0x7f52e8072bc0 msgr2=0x7f52e8072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 -- 172.21.15.100:0/1887180929 shutdown_connections 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.906+0000 7f52ee7de700 1 -- 172.21.15.100:0/1887180929 wait complete. 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.907+0000 7f52ee7de700 1 Processor -- start 2024-03-24T11:15:53.908 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.907+0000 7f52ee7de700 1 -- start start 2024-03-24T11:15:53.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.907+0000 7f52ee7de700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 0x7f52e8113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:53.909 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.907+0000 7f52ee7de700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e8114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.907+0000 7f52ee7de700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e8114730 con 0x7f52e8102710 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.907+0000 7f52ee7de700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52e810dad0 con 0x7f52e8101d70 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e8114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e8114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56776/0 (socket says 172.21.15.100:56776) 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 -- 172.21.15.100:0/684816689 learned_addr learned my addr 172.21.15.100:0/684816689 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e7fff700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 0x7f52e8113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 -- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 msgr2=0x7f52e8113b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 0x7f52e8113b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 -- 172.21.15.100:0/684816689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52d8005040 con 0x7f52e8102710 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e7fff700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 0x7f52e8113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e77fe700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e8114050 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009100 tx=0x7f52d8009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:53.910 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52d800e040 con 0x7f52e8102710 2024-03-24T11:15:53.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f52d8013420 con 0x7f52e8102710 2024-03-24T11:15:53.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52e810dd50 con 0x7f52e8102710 2024-03-24T11:15:53.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.908+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f52d801c610 con 0x7f52e8102710 2024-03-24T11:15:53.911 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.909+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52e810e310 con 0x7f52e8102710 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.910+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f52e810e880 con 0x7f52e8102710 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.910+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f52d8013590 con 0x7f52e8102710 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.910+0000 7f52e4ff9700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f52d40692a0 0x7f52d406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f52d8087ff0 con 0x7f52e8102710 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52e4ff9700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] conn(0x7f52d406ef70 0x7f52d4071380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52e7fff700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f52d40692a0 0x7f52d406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 --> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f52d4071a30 con 0x7f52d406ef70 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f52d808a660 con 0x7f52e8102710 2024-03-24T11:15:53.912 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52ecd7b700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] conn(0x7f52d406ef70 0x7f52d4071380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:53.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.911+0000 7f52e7fff700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f52d40692a0 0x7f52d406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009a80 tx=0x7f52d0008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:53.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.912+0000 7f52ecd7b700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] conn(0x7f52d406ef70 0x7f52d4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:53.913 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.912+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== osd.3 v2:172.21.15.105:6800/3221138489 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f52d4071a30 con 0x7f52d406ef70 2024-03-24T11:15:53.914 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:53.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.929+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 --> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f52e8105fe0 con 0x7f52d406ef70 2024-03-24T11:15:53.930 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.930+0000 7f52e4ff9700 1 -- 172.21.15.100:0/684816689 <== osd.3 v2:172.21.15.105:6800/3221138489 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f52e8105fe0 con 0x7f52d406ef70 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.930+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] conn(0x7f52d406ef70 msgr2=0x7f52d4071380 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.930+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] conn(0x7f52d406ef70 0x7f52d4071380 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f52d40692a0 msgr2=0x7f52d406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f52d40692a0 0x7f52d406b750 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f52d0009a80 tx=0x7f52d0008040 comp rx=0 tx=0).stop 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 msgr2=0x7f52e8114050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e8114050 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f52d8009100 tx=0x7f52d8009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 shutdown_connections 2024-03-24T11:15:53.932 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:6800/3221138489,v1:172.21.15.105:6801/3221138489] conn(0x7f52d406ef70 0x7f52d4071380 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f52d40692a0 0x7f52d406b750 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f52e8101d70 0x7f52e8113b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 --2- 172.21.15.100:0/684816689 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f52e8102710 0x7f52e8114050 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:53.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 >> 172.21.15.100:0/684816689 conn(0x7f52e8072bc0 msgr2=0x7f52e8065940 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:53.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.931+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 shutdown_connections 2024-03-24T11:15:53.933 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:53.932+0000 7f52ee7de700 1 -- 172.21.15.100:0/684816689 wait complete. 2024-03-24T11:15:54.181 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:54 smithi105 ceph-mon[36437]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:54.560 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:54 smithi100 ceph-mon[28941]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:56.051 INFO:teuthology.orchestra.run.smithi100.stdout:115964117013 2024-03-24T11:15:56.051 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph osd last-stat-seq osd.3 2024-03-24T11:15:56.400 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:56.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:56 smithi105 ceph-mon[36437]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:56.553 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:56 smithi100 ceph-mon[28941]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:58.021 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3992681934 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 msgr2=0x7ff1ec106f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3992681934 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec106f00 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7ff1d4004940 tx=0x7ff1d4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3992681934 shutdown_connections 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3992681934 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec106f00 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3992681934 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1ec0fde90 0x7ff1ec0fe260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3992681934 >> 172.21.15.100:0/3992681934 conn(0x7ff1ec072bc0 msgr2=0x7ff1ec072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3992681934 shutdown_connections 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3992681934 wait complete. 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.019+0000 7ff1f0f0c700 1 Processor -- start 2024-03-24T11:15:58.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 -- start start 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1ec0fde90 0x7ff1ec113b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec1140d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec1147b0 con 0x7ff1ec0fe830 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1ec10db10 con 0x7ff1ec0fde90 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec1140d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec1140d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56708/0 (socket says 172.21.15.100:56708) 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 -- 172.21.15.100:0/3449514085 learned_addr learned my addr 172.21.15.100:0/3449514085 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 -- 172.21.15.100:0/3449514085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1ec0fde90 msgr2=0x7ff1ec113b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1ea59c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1ec0fde90 0x7ff1ec113b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1ec0fde90 0x7ff1ec113b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 -- 172.21.15.100:0/3449514085 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d4005040 con 0x7ff1ec0fe830 2024-03-24T11:15:58.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e9d9b700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec1140d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400b3b0 tx=0x7ff1d4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:58.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1d400e040 con 0x7ff1ec0fe830 2024-03-24T11:15:58.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff1d4013420 con 0x7ff1ec0fe830 2024-03-24T11:15:58.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1ec10dd90 con 0x7ff1ec0fe830 2024-03-24T11:15:58.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff1d401c610 con 0x7ff1ec0fe830 2024-03-24T11:15:58.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.020+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1ec10e2d0 con 0x7ff1ec0fe830 2024-03-24T11:15:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.022+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1ec065e30 con 0x7ff1ec0fe830 2024-03-24T11:15:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.022+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff1d4013590 con 0x7ff1ec0fe830 2024-03-24T11:15:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.022+0000 7ff1e2ffd700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff1d80692a0 0x7ff1d806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.023+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff1d4088000 con 0x7ff1ec0fe830 2024-03-24T11:15:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.023+0000 7ff1ea59c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff1d80692a0 0x7ff1d806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:58.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.023+0000 7ff1ea59c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff1d80692a0 0x7ff1d806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc007f30 tx=0x7ff1dc007420 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:58.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.028+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff1d40581a0 con 0x7ff1ec0fe830 2024-03-24T11:15:58.033 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:58.033 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:15:58.226 INFO:teuthology.orchestra.run.smithi100.stdout:81604378649 2024-03-24T11:15:58.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.224+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7ff1ec05f5f0 con 0x7ff1ec0fe830 2024-03-24T11:15:58.226 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.225+0000 7ff1e2ffd700 1 -- 172.21.15.100:0/3449514085 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7ff1d4013840 con 0x7ff1ec0fe830 2024-03-24T11:15:58.227 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff1d80692a0 msgr2=0x7ff1d806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff1d80692a0 0x7ff1d806b750 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7ff1dc007f30 tx=0x7ff1dc007420 comp rx=0 tx=0).stop 2024-03-24T11:15:58.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 msgr2=0x7ff1ec1140d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec1140d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7ff1d400b3b0 tx=0x7ff1d4009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:58.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 shutdown_connections 2024-03-24T11:15:58.228 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff1d80692a0 0x7ff1d806b750 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff1ec0fde90 0x7ff1ec113b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 --2- 172.21.15.100:0/3449514085 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff1ec0fe830 0x7ff1ec1140d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 >> 172.21.15.100:0/3449514085 conn(0x7ff1ec072bc0 msgr2=0x7ff1ec103740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:58.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 shutdown_connections 2024-03-24T11:15:58.229 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.227+0000 7ff1f0f0c700 1 -- 172.21.15.100:0/3449514085 wait complete. 2024-03-24T11:15:58.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:58 smithi100 ceph-mon[28941]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:58.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:58 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3449514085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:15:58.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:58 smithi105 ceph-mon[36437]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:15:58.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:58 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3449514085' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-03-24T11:15:58.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.716+0000 7f2e2029a700 1 -- 172.21.15.100:0/2695792338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 msgr2=0x7f2e181031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.716+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2695792338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e181031d0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7f2e14004940 tx=0x7f2e14009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:58.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.716+0000 7f2e2029a700 1 -- 172.21.15.100:0/2695792338 shutdown_connections 2024-03-24T11:15:58.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2695792338 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e181037a0 0x7f2e180fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2695792338 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e181031d0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.719 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 -- 172.21.15.100:0/2695792338 >> 172.21.15.100:0/2695792338 conn(0x7f2e180f89b0 msgr2=0x7f2e180fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 -- 172.21.15.100:0/2695792338 shutdown_connections 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 -- 172.21.15.100:0/2695792338 wait complete. 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 Processor -- start 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 -- start start 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e18113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e181037a0 0x7f2e18113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18114550 con 0x7f2e18102e00 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e2029a700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e1810dac0 con 0x7f2e181037a0 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e1e036700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e18113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e1e036700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e18113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56728/0 (socket says 172.21.15.100:56728) 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e1e036700 1 -- 172.21.15.100:0/2224163341 learned_addr learned my addr 172.21.15.100:0/2224163341 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:15:58.720 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.717+0000 7f2e1d835700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e181037a0 0x7f2e18113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e1e036700 1 -- 172.21.15.100:0/2224163341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e181037a0 msgr2=0x7f2e18113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e1e036700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e181037a0 0x7f2e18113e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e1e036700 1 -- 172.21.15.100:0/2224163341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e14005040 con 0x7f2e18102e00 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e1e036700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e18113930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400b3b0 tx=0x7f2e14009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e1400e040 con 0x7f2e18102e00 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e14013420 con 0x7f2e18102e00 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e1810dd40 con 0x7f2e18102e00 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e1401c630 con 0x7f2e18102e00 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.718+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e1810e300 con 0x7f2e18102e00 2024-03-24T11:15:58.721 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.719+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e180fe550 con 0x7f2e18102e00 2024-03-24T11:15:58.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.719+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2e14013590 con 0x7f2e18102e00 2024-03-24T11:15:58.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.720+0000 7f2e0effd700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e040692a0 0x7f2e0406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:15:58.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.720+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2e14088030 con 0x7f2e18102e00 2024-03-24T11:15:58.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.720+0000 7f2e1d835700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e040692a0 0x7f2e0406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:15:58.722 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.720+0000 7f2e1d835700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e040692a0 0x7f2e0406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e1810ec80 tx=0x7f2e080091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:15:58.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.724+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2e14058130 con 0x7f2e18102e00 2024-03-24T11:15:58.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.915+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f2e18109f70 con 0x7f2e18102e00 2024-03-24T11:15:58.916 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.915+0000 7f2e0effd700 1 -- 172.21.15.100:0/2224163341 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f2e14057cc0 con 0x7f2e18102e00 2024-03-24T11:15:58.916 INFO:teuthology.orchestra.run.smithi100.stdout:150323855376 2024-03-24T11:15:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e040692a0 msgr2=0x7f2e0406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e040692a0 0x7f2e0406b750 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f2e1810ec80 tx=0x7f2e080091a0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 msgr2=0x7f2e18113930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:15:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e18113930 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f2e1400b3b0 tx=0x7f2e14009e30 comp rx=0 tx=0).stop 2024-03-24T11:15:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 shutdown_connections 2024-03-24T11:15:58.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e040692a0 0x7f2e0406b750 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e18102e00 0x7f2e18113930 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 --2- 172.21.15.100:0/2224163341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e181037a0 0x7f2e18113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:15:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 >> 172.21.15.100:0/2224163341 conn(0x7f2e180f89b0 msgr2=0x7f2e180f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:15:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.917+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 shutdown_connections 2024-03-24T11:15:58.919 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:15:58.918+0000 7f2e2029a700 1 -- 172.21.15.100:0/2224163341 wait complete. 2024-03-24T11:15:59.246 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378649 for osd.2 2024-03-24T11:15:59.246 DEBUG:teuthology.parallel:result is None 2024-03-24T11:15:59.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:15:59 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2224163341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:15:59.575 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:15:59 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2224163341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-03-24T11:15:59.893 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855374 got 150323855376 for osd.4 2024-03-24T11:15:59.893 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:00.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:00 smithi100 ceph-mon[28941]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:00.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:00 smithi105 ceph-mon[36437]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:02.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:02 smithi105 ceph-mon[36437]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:02.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:02 smithi100 ceph-mon[28941]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 -- 172.21.15.100:0/281167462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 msgr2=0x7f0cfc106df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 --2- 172.21.15.100:0/281167462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc106df0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f0ce4004940 tx=0x7f0ce4009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 -- 172.21.15.100:0/281167462 shutdown_connections 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 --2- 172.21.15.100:0/281167462 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc106df0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 --2- 172.21.15.100:0/281167462 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cfc0fde10 0x7f0cfc0fe1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 -- 172.21.15.100:0/281167462 >> 172.21.15.100:0/281167462 conn(0x7f0cfc0f7880 msgr2=0x7f0cfc0f9c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 -- 172.21.15.100:0/281167462 shutdown_connections 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 -- 172.21.15.100:0/281167462 wait complete. 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 Processor -- start 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.821+0000 7f0d02e18700 1 -- start start 2024-03-24T11:16:02.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d02e18700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cfc0fde10 0x7f0cfc0fbc40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d02e18700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc0fc180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d02e18700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0fc710 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d02e18700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cfc0fc850 con 0x7f0cfc0fde10 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc0fc180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc0fc180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56740/0 (socket says 172.21.15.100:56740) 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 -- 172.21.15.100:0/390064276 learned_addr learned my addr 172.21.15.100:0/390064276 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d00bb4700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cfc0fde10 0x7f0cfc0fbc40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 -- 172.21.15.100:0/390064276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cfc0fde10 msgr2=0x7f0cfc0fbc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cfc0fde10 0x7f0cfc0fbc40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 -- 172.21.15.100:0/390064276 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ce4005040 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cfbfff700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc0fc180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400b3b0 tx=0x7f0ce4009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce400e040 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0ce4013420 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.824 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0ce401c630 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cfc0fcaa0 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.822+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cfc11a550 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.825 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.823+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cfc066770 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.824+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0ce4013590 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.824+0000 7f0cf97fa700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0ce80692b0 0x7f0ce806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:02.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.824+0000 7f0d00bb4700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0ce80692b0 0x7f0ce806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:02.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.824+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0ce4087f90 con 0x7f0cfc0fe7b0 2024-03-24T11:16:02.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.825+0000 7f0d00bb4700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0ce80692b0 0x7f0ce806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0cec004550 tx=0x7f0cec007ed0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:02.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:02.829+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0ce40546a0 con 0x7f0cfc0fe7b0 2024-03-24T11:16:03.020 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.019+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f0cfc04bed0 con 0x7f0cfc0fe7b0 2024-03-24T11:16:03.020 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.019+0000 7f0cf97fa700 1 -- 172.21.15.100:0/390064276 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0ce4057cc0 con 0x7f0cfc0fe7b0 2024-03-24T11:16:03.020 INFO:teuthology.orchestra.run.smithi100.stdout:60129542177 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.021+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0ce80692b0 msgr2=0x7f0ce806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.021+0000 7f0d02e18700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0ce80692b0 0x7f0ce806b760 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7f0cec004550 tx=0x7f0cec007ed0 comp rx=0 tx=0).stop 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.021+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 msgr2=0x7f0cfc0fc180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.021+0000 7f0d02e18700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc0fc180 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7f0ce400b3b0 tx=0x7f0ce4009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 shutdown_connections 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0ce80692b0 0x7f0ce806b760 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cfc0fde10 0x7f0cfc0fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:03.022 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 --2- 172.21.15.100:0/390064276 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0cfc0fe7b0 0x7f0cfc0fc180 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:03.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 >> 172.21.15.100:0/390064276 conn(0x7f0cfc0f7880 msgr2=0x7f0cfc0f8e00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:03.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 shutdown_connections 2024-03-24T11:16:03.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:03.022+0000 7f0d02e18700 1 -- 172.21.15.100:0/390064276 wait complete. 2024-03-24T11:16:03.585 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:03 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/390064276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:16:03.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:03 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/390064276' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-03-24T11:16:03.976 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:04.001 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542174 got 60129542177 for osd.1 2024-03-24T11:16:04.002 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:04.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:04 smithi100 ceph-mon[28941]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:04.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:04 smithi105 ceph-mon[36437]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:04.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 -- 172.21.15.100:0/547300805 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c8101d30 msgr2=0x7fb0c8102100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:04.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 --2- 172.21.15.100:0/547300805 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c8101d30 0x7fb0c8102100 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4004940 tx=0x7fb0c4009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:04.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 -- 172.21.15.100:0/547300805 shutdown_connections 2024-03-24T11:16:04.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 --2- 172.21.15.100:0/547300805 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0c81026d0 0x7fb0c80fbd40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:04.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 --2- 172.21.15.100:0/547300805 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c8101d30 0x7fb0c8102100 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:04.866 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 -- 172.21.15.100:0/547300805 >> 172.21.15.100:0/547300805 conn(0x7fb0c80f78c0 msgr2=0x7fb0c80f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 -- 172.21.15.100:0/547300805 shutdown_connections 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 -- 172.21.15.100:0/547300805 wait complete. 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.850+0000 7fb0cf436700 1 Processor -- start 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cf436700 1 -- start start 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cf436700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0c8101d30 0x7fb0c8115b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cf436700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 0x7fb0c8116080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cf436700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c81166d0 con 0x7fb0c81026d0 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cf436700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0c810fcb0 con 0x7fb0c8101d30 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 0x7fb0c8116080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 0x7fb0c8116080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56746/0 (socket says 172.21.15.100:56746) 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 -- 172.21.15.100:0/3839631000 learned_addr learned my addr 172.21.15.100:0/3839631000 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cd1d2700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0c8101d30 0x7fb0c8115b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 -- 172.21.15.100:0/3839631000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0c8101d30 msgr2=0x7fb0c8115b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0c8101d30 0x7fb0c8115b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:04.867 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 -- 172.21.15.100:0/3839631000 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0c4005040 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cc9d1700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 0x7fb0c8116080 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8002a80 tx=0x7fb0b8004630 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0b8004e80 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.851+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0c810ff90 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.852+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb0b800f040 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.852+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb0b800daa0 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.852+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0c8110550 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.853+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0c8105950 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.853+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb0b800dc00 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.853+0000 7fb0bdffb700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0b40692a0 0x7fb0b406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.853+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb0b80863e0 con 0x7fb0c81026d0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.853+0000 7fb0cd1d2700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0b40692a0 0x7fb0b406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.854+0000 7fb0cd1d2700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0b40692a0 0x7fb0b406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4004f90 tx=0x7fb0c4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:04.868 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:04.858+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb0b80564e0 con 0x7fb0c81026d0 2024-03-24T11:16:05.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.049+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fb0c8111190 con 0x7fb0c81026d0 2024-03-24T11:16:05.050 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.050+0000 7fb0bdffb700 1 -- 172.21.15.100:0/3839631000 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fb0b8056070 con 0x7fb0c81026d0 2024-03-24T11:16:05.050 INFO:teuthology.orchestra.run.smithi100.stdout:171798691852 2024-03-24T11:16:05.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0b40692a0 msgr2=0x7fb0b406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:05.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0b40692a0 0x7fb0b406b750 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fb0c4004f90 tx=0x7fb0c4009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:05.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 msgr2=0x7fb0c8116080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:05.052 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 0x7fb0c8116080 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7fb0b8002a80 tx=0x7fb0b8004630 comp rx=0 tx=0).stop 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 shutdown_connections 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb0b40692a0 0x7fb0b406b750 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb0c8101d30 0x7fb0c8115b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 --2- 172.21.15.100:0/3839631000 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb0c81026d0 0x7fb0c8116080 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 >> 172.21.15.100:0/3839631000 conn(0x7fb0c80f78c0 msgr2=0x7fb0c81082b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 shutdown_connections 2024-03-24T11:16:05.053 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:05.052+0000 7fb0cf436700 1 -- 172.21.15.100:0/3839631000 wait complete. 2024-03-24T11:16:05.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:05 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3839631000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:16:05.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:05 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3839631000' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.238+0000 7fb1a81db700 1 -- 172.21.15.100:0/191147127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 msgr2=0x7fb1a01031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.238+0000 7fb1a81db700 1 --2- 172.21.15.100:0/191147127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a01031d0 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7fb19c004940 tx=0x7fb19c009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.238+0000 7fb1a81db700 1 -- 172.21.15.100:0/191147127 shutdown_connections 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.238+0000 7fb1a81db700 1 --2- 172.21.15.100:0/191147127 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a01037a0 0x7fb1a00fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.238+0000 7fb1a81db700 1 --2- 172.21.15.100:0/191147127 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a01031d0 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.238+0000 7fb1a81db700 1 -- 172.21.15.100:0/191147127 >> 172.21.15.100:0/191147127 conn(0x7fb1a00f89b0 msgr2=0x7fb1a00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 -- 172.21.15.100:0/191147127 shutdown_connections 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 -- 172.21.15.100:0/191147127 wait complete. 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 Processor -- start 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 -- start start 2024-03-24T11:16:06.241 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a0113930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a01037a0 0x7fb1a0113e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a0114550 con 0x7fb1a0102e00 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a81db700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1a010dac0 con 0x7fb1a01037a0 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a5f77700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a0113930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a5f77700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a0113930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56772/0 (socket says 172.21.15.100:56772) 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.239+0000 7fb1a5f77700 1 -- 172.21.15.100:0/1857764434 learned_addr learned my addr 172.21.15.100:0/1857764434 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1a5f77700 1 -- 172.21.15.100:0/1857764434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a01037a0 msgr2=0x7fb1a0113e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1a5f77700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a01037a0 0x7fb1a0113e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1a5f77700 1 -- 172.21.15.100:0/1857764434 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb19c005040 con 0x7fb1a0102e00 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1a5f77700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a0113930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00b3b0 tx=0x7fb19c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb19c00e040 con 0x7fb1a0102e00 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb19c00bbc0 con 0x7fb1a0102e00 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1a010dd40 con 0x7fb1a0102e00 2024-03-24T11:16:06.242 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb19c01c3f0 con 0x7fb1a0102e00 2024-03-24T11:16:06.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.240+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1a010e300 con 0x7fb1a0102e00 2024-03-24T11:16:06.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.241+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1a00fe550 con 0x7fb1a0102e00 2024-03-24T11:16:06.243 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.241+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb19c00bd30 con 0x7fb1a0102e00 2024-03-24T11:16:06.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.242+0000 7fb1967fc700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb18c0692a0 0x7fb18c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:06.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.242+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb19c088020 con 0x7fb1a0102e00 2024-03-24T11:16:06.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.242+0000 7fb1a5776700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb18c0692a0 0x7fb18c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:06.244 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.242+0000 7fb1a5776700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb18c0692a0 0x7fb18c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb190006fd0 tx=0x7fb190006db0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:06.246 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.246+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb19c0581d0 con 0x7fb1a0102e00 2024-03-24T11:16:06.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.435+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7fb1a0109fd0 con 0x7fb1a0102e00 2024-03-24T11:16:06.436 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.435+0000 7fb1967fc700 1 -- 172.21.15.100:0/1857764434 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fb19c057d60 con 0x7fb1a0102e00 2024-03-24T11:16:06.436 INFO:teuthology.orchestra.run.smithi100.stdout:38654705702 2024-03-24T11:16:06.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.437+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb18c0692a0 msgr2=0x7fb18c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.437+0000 7fb1a81db700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb18c0692a0 0x7fb18c06b750 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7fb190006fd0 tx=0x7fb190006db0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.437+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 msgr2=0x7fb1a0113930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.438 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.437+0000 7fb1a81db700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a0113930 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7fb19c00b3b0 tx=0x7fb19c009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 shutdown_connections 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb18c0692a0 0x7fb18c06b750 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1a0102e00 0x7fb1a0113930 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 --2- 172.21.15.100:0/1857764434 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1a01037a0 0x7fb1a0113e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 >> 172.21.15.100:0/1857764434 conn(0x7fb1a00f89b0 msgr2=0x7fb1a00f98b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 shutdown_connections 2024-03-24T11:16:06.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.438+0000 7fb1a81db700 1 -- 172.21.15.100:0/1857764434 wait complete. 2024-03-24T11:16:06.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 -- 172.21.15.100:0/4212804883 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c102e70 msgr2=0x7fac3c103240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 --2- 172.21.15.100:0/4212804883 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c102e70 0x7fac3c103240 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7fac30004940 tx=0x7fac30009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:06.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 -- 172.21.15.100:0/4212804883 shutdown_connections 2024-03-24T11:16:06.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 --2- 172.21.15.100:0/4212804883 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fb910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 --2- 172.21.15.100:0/4212804883 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c102e70 0x7fac3c103240 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 -- 172.21.15.100:0/4212804883 >> 172.21.15.100:0/4212804883 conn(0x7fac3c0714c0 msgr2=0x7fac3c0718c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:06.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 -- 172.21.15.100:0/4212804883 shutdown_connections 2024-03-24T11:16:06.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.469+0000 7fac43512700 1 -- 172.21.15.100:0/4212804883 wait complete. 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac43512700 1 Processor -- start 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac43512700 1 -- start start 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac43512700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fc680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac43512700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac3c102e70 0x7fac3c0fcbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac43512700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c0fd100 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac43512700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac3c0fd240 con 0x7fac3c102e70 2024-03-24T11:16:06.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac412ae700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fc680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac412ae700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fc680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:56786/0 (socket says 172.21.15.100:56786) 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac412ae700 1 -- 172.21.15.100:0/3086043613 learned_addr learned my addr 172.21.15.100:0/3086043613 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac40aad700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac3c102e70 0x7fac3c0fcbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac412ae700 1 -- 172.21.15.100:0/3086043613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac3c102e70 msgr2=0x7fac3c0fcbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac412ae700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac3c102e70 0x7fac3c0fcbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.470+0000 7fac412ae700 1 -- 172.21.15.100:0/3086043613 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac30005040 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.471+0000 7fac412ae700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fc680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fac3000b0d0 tx=0x7fac30009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.471+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac30021030 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.471+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fac3001c400 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.471+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac3000e070 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.471+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac3c0fd490 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.472+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac3c11a020 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.472+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac3c100110 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.473+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fac30024070 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.473+0000 7fac2dffb700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fac280690a0 0x7fac2806b550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:06.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.473+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fac30022070 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.473+0000 7fac40aad700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fac280690a0 0x7fac2806b550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:06.475 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.474+0000 7fac40aad700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fac280690a0 0x7fac2806b550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0fdf70 tx=0x7fac3800b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:06.478 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.478+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fac30053e50 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:06 smithi100 ceph-mon[28941]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:06.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.666+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fac3c04bed0 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.667 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.666+0000 7fac2dffb700 1 -- 172.21.15.100:0/3086043613 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fac30057470 con 0x7fac3c0fb4a0 2024-03-24T11:16:06.667 INFO:teuthology.orchestra.run.smithi100.stdout:115964117015 2024-03-24T11:16:06.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fac280690a0 msgr2=0x7fac2806b550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.669 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fac280690a0 0x7fac2806b550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fac3c0fdf70 tx=0x7fac3800b040 comp rx=0 tx=0).stop 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 msgr2=0x7fac3c0fc680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fc680 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fac3000b0d0 tx=0x7fac30009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 shutdown_connections 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fac280690a0 0x7fac2806b550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fac3c0fb4a0 0x7fac3c0fc680 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 --2- 172.21.15.100:0/3086043613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac3c102e70 0x7fac3c0fcbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 >> 172.21.15.100:0/3086043613 conn(0x7fac3c0714c0 msgr2=0x7fac3c0fa690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 shutdown_connections 2024-03-24T11:16:06.670 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:06.669+0000 7fac43512700 1 -- 172.21.15.100:0/3086043613 wait complete. 2024-03-24T11:16:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:06 smithi105 ceph-mon[36437]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:07.105 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705702 for osd.0 2024-03-24T11:16:07.106 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:07.339 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1857764434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:16:07.339 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3086043613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:16:07.703 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117013 got 115964117015 for osd.3 2024-03-24T11:16:07.703 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:07 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/1857764434' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-03-24T11:16:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:07 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3086043613' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-03-24T11:16:08.149 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691850 got 171798691852 for osd.5 2024-03-24T11:16:08.149 DEBUG:teuthology.parallel:result is None 2024-03-24T11:16:08.150 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-03-24T11:16:08.150 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:16:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:08 smithi100 ceph-mon[28941]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:08.414 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:08.703 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:08 smithi105 ceph-mon[36437]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:08.704 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 -- 172.21.15.100:0/1835930563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc1026c0 msgr2=0x7f3ddc0fbd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 --2- 172.21.15.100:0/1835930563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc1026c0 0x7f3ddc0fbd30 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0004940 tx=0x7f3dd0009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 -- 172.21.15.100:0/1835930563 shutdown_connections 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 --2- 172.21.15.100:0/1835930563 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc1026c0 0x7f3ddc0fbd30 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 --2- 172.21.15.100:0/1835930563 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ddc101d20 0x7f3ddc1020f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 -- 172.21.15.100:0/1835930563 >> 172.21.15.100:0/1835930563 conn(0x7f3ddc0f7870 msgr2=0x7f3ddc0f9c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 -- 172.21.15.100:0/1835930563 shutdown_connections 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.195+0000 7f3de0abb700 1 -- 172.21.15.100:0/1835930563 wait complete. 2024-03-24T11:16:10.200 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3de0abb700 1 Processor -- start 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3de0abb700 1 -- start start 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3de0abb700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 0x7f3ddc10eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3de0abb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ddc1026c0 0x7f3ddc10f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3de0abb700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc10f5c0 con 0x7f3ddc101d20 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3de0abb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ddc10f700 con 0x7f3ddc1026c0 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 0x7f3ddc10eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 0x7f3ddc10eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53968/0 (socket says 172.21.15.100:53968) 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 -- 172.21.15.100:0/2206046667 learned_addr learned my addr 172.21.15.100:0/2206046667 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dd9d9b700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ddc1026c0 0x7f3ddc10f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 -- 172.21.15.100:0/2206046667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ddc1026c0 msgr2=0x7f3ddc10f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ddc1026c0 0x7f3ddc10f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 -- 172.21.15.100:0/2206046667 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd0005040 con 0x7f3ddc101d20 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dda59c700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 0x7f3ddc10eb40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b790 tx=0x7f3dcc009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.196+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dcc00f6f0 con 0x7f3ddc101d20 2024-03-24T11:16:10.201 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.197+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f3dcc00fd20 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.197+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3dcc0103a0 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.197+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ddc10f990 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.197+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ddc117fb0 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.198+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ddc0fd470 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.198+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f3dcc011400 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.198+0000 7f3dcaffd700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3dc40692a0 0x7f3dc406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.198+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3dcc086c30 con 0x7f3ddc101d20 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.199+0000 7f3dd9d9b700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3dc40692a0 0x7f3dc406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:10.202 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.199+0000 7f3dd9d9b700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3dc40692a0 0x7f3dc406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0004890 tx=0x7f3dd0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:10.203 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.203+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3dcc0532a0 con 0x7f3ddc101d20 2024-03-24T11:16:10.386 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.386+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f3ddc0668f0 con 0x7f3dc40692a0 2024-03-24T11:16:10.387 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:10 smithi100 ceph-mon[28941]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:10.387 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.387+0000 7f3dcaffd700 1 -- 172.21.15.100:0/2206046667 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28140 (secure 0 0 0) 0x7f3ddc0668f0 con 0x7f3dc40692a0 2024-03-24T11:16:10.388 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3dc40692a0 msgr2=0x7f3dc406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3dc40692a0 0x7f3dc406b750 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f3dd0004890 tx=0x7f3dd0009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 msgr2=0x7f3ddc10eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 0x7f3ddc10eb40 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f3dcc00b790 tx=0x7f3dcc009250 comp rx=0 tx=0).stop 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 shutdown_connections 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f3dc40692a0 0x7f3dc406b750 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f3ddc101d20 0x7f3ddc10eb40 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 --2- 172.21.15.100:0/2206046667 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ddc1026c0 0x7f3ddc10f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:10.390 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.389+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 >> 172.21.15.100:0/2206046667 conn(0x7f3ddc0f7870 msgr2=0x7f3ddc0f87a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:10.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.390+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 shutdown_connections 2024-03-24T11:16:10.391 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:10.390+0000 7f3de0abb700 1 -- 172.21.15.100:0/2206046667 wait complete. 2024-03-24T11:16:10.391 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-03-24T11:16:10.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:10 smithi105 ceph-mon[36437]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:11.055 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-03-24T11:16:09.148147+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002644"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-24T11:15:17.866563+0000","last_change":"2024-03-24T11:14:51.638811+0000","last_active":"2024-03-24T11:15:17.866563+0000","last_peered":"2024-03-24T11:15:17.866563+0000","last_clean":"2024-03-24T11:15:17.866563+0000","last_became_active":"2024-03-24T11:14:51.638488+0000","last_became_peered":"2024-03-24T11:14:51.638488+0000","last_unstale":"2024-03-24T11:15:17.866563+0000","last_undegraded":"2024-03-24T11:15:17.866563+0000","last_fullsized":"2024-03-24T11:15:17.866563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:14:05.668365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:14:05.668365+0000","last_clean_scrub_stamp":"2024-03-24T11:14:05.668365+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:30.770974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.624},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.59399999999999997}]},{"osd":1,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.57999999999999996},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.50900000000000001}]},{"osd":2,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.55000000000000004},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.495}]},{"osd":3,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.34499999999999997},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44700000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40300000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117016,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.433},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.35099999999999998}]},{"osd":1,"last update":"Sun Mar 24 11:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.47799999999999998}]},{"osd":2,"last update":"Sun Mar 24 11:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.48999999999999999}]},{"osd":4,"last update":"Sun Mar 24 11:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.38900000000000001},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.45400000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51800000000000002}]}]},{"osd":2,"up_from":19,"seq":81604378651,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"min":{"1min":0.254,"5min":0.254,"15min":0.254},"max":{"1min":0.504,"5min":0.504,"15min":0.504},"last":0.34599999999999997},{"interface":"front","average":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"min":{"1min":0.249,"5min":0.249,"15min":0.249},"max":{"1min":0.48199999999999998,"5min":0.48199999999999998,"15min":0.48199999999999998},"last":0.32800000000000001}]},{"osd":1,"last update":"Sun Mar 24 11:15:03 2024","interfaces":[{"interface":"back","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"min":{"1min":0.28799999999999998,"5min":0.28799999999999998,"15min":0.28799999999999998},"max":{"1min":0.56499999999999995,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.376}]},{"osd":3,"last update":"Sun Mar 24 11:15:28 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.627}]},{"osd":4,"last update":"Sun Mar 24 11:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.65400000000000003},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.64200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60099999999999998}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 11:15:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.314},"min":{"1min":0.21199999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.36899999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.33500000000000002,"15min":0.32900000000000001},"min":{"1min":0.23100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.307}]},{"osd":2,"last update":"Sun Mar 24 11:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.35499999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:15:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.48299999999999998},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.46800000000000003}]},{"osd":4,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.40200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44500000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542178,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23999999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.25},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.34200000000000003,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.29099999999999998}]},{"osd":2,"last update":"Sun Mar 24 11:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":0.57799999999999996,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.312},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.59899999999999998,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.26800000000000002}]},{"osd":3,"last update":"Sun Mar 24 11:15:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.64200000000000002},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.60899999999999999}]},{"osd":4,"last update":"Sun Mar 24 11:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.63100000000000001},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.59899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.624}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:16:11.056 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph pg dump --format=json 2024-03-24T11:16:11.311 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:11.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:11 smithi100 ceph-mon[28941]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:16:11.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:11 smithi105 ceph-mon[36437]: from='client.14428 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:16:12.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:12 smithi100 ceph-mon[28941]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:12.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:12 smithi105 ceph-mon[36437]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:13.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 -- 172.21.15.100:0/822808010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4102710 msgr2=0x7f07b40fbd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:13.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/822808010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4102710 0x7f07b40fbd80 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f079c004940 tx=0x7f079c009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:13.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 -- 172.21.15.100:0/822808010 shutdown_connections 2024-03-24T11:16:13.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/822808010 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4102710 0x7f07b40fbd80 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.023 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/822808010 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07b4101d70 0x7f07b4102140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 -- 172.21.15.100:0/822808010 >> 172.21.15.100:0/822808010 conn(0x7f07b4072bc0 msgr2=0x7f07b4072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:13.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 -- 172.21.15.100:0/822808010 shutdown_connections 2024-03-24T11:16:13.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.017+0000 7f07b8f4c700 1 -- 172.21.15.100:0/822808010 wait complete. 2024-03-24T11:16:13.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 Processor -- start 2024-03-24T11:16:13.024 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 -- start start 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 0x7f07b410ece0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07b4102710 0x7f07b410f220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b410f760 con 0x7f07b4101d70 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b410f8a0 con 0x7f07b4102710 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 0x7f07b410ece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 0x7f07b410ece0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53986/0 (socket says 172.21.15.100:53986) 2024-03-24T11:16:13.025 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 -- 172.21.15.100:0/1739467755 learned_addr learned my addr 172.21.15.100:0/1739467755 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:13.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b1d9b700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07b4102710 0x7f07b410f220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:13.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 -- 172.21.15.100:0/1739467755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07b4102710 msgr2=0x7f07b410f220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:13.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07b4102710 0x7f07b410f220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 -- 172.21.15.100:0/1739467755 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079c005040 con 0x7f07b4101d70 2024-03-24T11:16:13.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b259c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 0x7f07b410ece0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f07ac001d80 tx=0x7f07ac009ec0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:13.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ac007d10 con 0x7f07b4101d70 2024-03-24T11:16:13.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f07ac001e80 con 0x7f07b4101d70 2024-03-24T11:16:13.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b4117b90 con 0x7f07b4101d70 2024-03-24T11:16:13.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f07ac014640 con 0x7f07b4101d70 2024-03-24T11:16:13.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.018+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b4118120 con 0x7f07b4101d70 2024-03-24T11:16:13.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.019+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b4104240 con 0x7f07b4101d70 2024-03-24T11:16:13.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.020+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f07ac00e960 con 0x7f07b4101d70 2024-03-24T11:16:13.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.020+0000 7f07aaffd700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f07a00692a0 0x7f07a006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:13.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.020+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f07ac086c10 con 0x7f07b4101d70 2024-03-24T11:16:13.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.020+0000 7f07b1d9b700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f07a00692a0 0x7f07a006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:13.028 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.021+0000 7f07b1d9b700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f07a00692a0 0x7f07a006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f079c004890 tx=0x7f079c009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:13.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.025+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f07ac04e190 con 0x7f07b4101d70 2024-03-24T11:16:13.208 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.206+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f07b404bed0 con 0x7f07a00692a0 2024-03-24T11:16:13.209 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.208+0000 7f07aaffd700 1 -- 172.21.15.100:0/1739467755 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+28211 (secure 0 0 0) 0x7f07b404bed0 con 0x7f07a00692a0 2024-03-24T11:16:13.210 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:13.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f07a00692a0 msgr2=0x7f07a006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:13.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f07a00692a0 0x7f07a006b750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f079c004890 tx=0x7f079c009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:13.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 msgr2=0x7f07b410ece0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:13.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 0x7f07b410ece0 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f07ac001d80 tx=0x7f07ac009ec0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 shutdown_connections 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f07a00692a0 0x7f07a006b750 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f07b4101d70 0x7f07b410ece0 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 --2- 172.21.15.100:0/1739467755 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07b4102710 0x7f07b410f220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 >> 172.21.15.100:0/1739467755 conn(0x7f07b4072bc0 msgr2=0x7f07b40fb080 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 shutdown_connections 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:13.210+0000 7f07b8f4c700 1 -- 172.21.15.100:0/1739467755 wait complete. 2024-03-24T11:16:13.212 INFO:teuthology.orchestra.run.smithi100.stderr:dumped all 2024-03-24T11:16:13.810 INFO:teuthology.orchestra.run.smithi100.stdout:{"pg_ready":true,"pg_map":{"version":159,"stamp":"2024-03-24T11:16:13.148779+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1787652,"kb_used_data":3924,"kb_used_omap":0,"kb_used_meta":1783680,"kb_avail":560683260,"statfs":{"total":575970213888,"available":574139658240,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":1826488320},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002460"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-03-24T11:15:17.866563+0000","last_change":"2024-03-24T11:14:51.638811+0000","last_active":"2024-03-24T11:15:17.866563+0000","last_peered":"2024-03-24T11:15:17.866563+0000","last_clean":"2024-03-24T11:15:17.866563+0000","last_became_active":"2024-03-24T11:14:51.638488+0000","last_became_peered":"2024-03-24T11:14:51.638488+0000","last_unstale":"2024-03-24T11:15:17.866563+0000","last_undegraded":"2024-03-24T11:15:17.866563+0000","last_fullsized":"2024-03-24T11:15:17.866563+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-03-24T11:14:05.668365+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-03-24T11:14:05.668365+0000","last_clean_scrub_stamp":"2024-03-24T11:14:05.668365+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-03-25T12:08:30.770974+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.378}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47899999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34200000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32600000000000001}]}]},{"osd":4,"up_from":35,"seq":150323855379,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.748,"5min":0.748,"15min":0.748},"last":0.69199999999999995},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.58399999999999996,"15min":0.58399999999999996},"min":{"1min":0.379,"5min":0.379,"15min":0.379},"max":{"1min":0.86199999999999999,"5min":0.86199999999999999,"15min":0.86199999999999999},"last":0.71099999999999997}]},{"osd":1,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.34100000000000003,"5min":0.34100000000000003,"15min":0.34100000000000003},"max":{"1min":0.92300000000000004,"5min":0.92300000000000004,"15min":0.92300000000000004},"last":0.63500000000000001},{"interface":"front","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.307,"5min":0.307,"15min":0.307},"max":{"1min":0.77200000000000002,"5min":0.77200000000000002,"15min":0.77200000000000002},"last":0.67200000000000004}]},{"osd":2,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.504,"5min":0.504,"15min":0.504},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.69999999999999996},"last":0.61799999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.88100000000000001,"5min":0.88100000000000001,"15min":0.88100000000000001},"last":0.59899999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.376,"5min":0.376,"15min":0.376},"min":{"1min":0.27800000000000002,"5min":0.27800000000000002,"15min":0.27800000000000002},"max":{"1min":0.56000000000000005,"5min":0.56000000000000005,"15min":0.56000000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"min":{"1min":0.26900000000000002,"5min":0.26900000000000002,"15min":0.26900000000000002},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.57799999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.64500000000000002}]}]},{"osd":3,"up_from":27,"seq":115964117017,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48599999999999999,"15min":0.48599999999999999},"min":{"1min":0.36099999999999999,"5min":0.36099999999999999,"15min":0.36099999999999999},"max":{"1min":0.61899999999999999,"5min":0.61899999999999999,"15min":0.61899999999999999},"last":0.33800000000000002},{"interface":"front","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.27200000000000002,"5min":0.27200000000000002,"15min":0.27200000000000002},"max":{"1min":0.64000000000000001,"5min":0.64000000000000001,"15min":0.64000000000000001},"last":0.32500000000000001}]},{"osd":1,"last update":"Sun Mar 24 11:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.628,"5min":0.628,"15min":0.628},"last":0.36299999999999999},{"interface":"front","average":{"1min":0.49199999999999999,"5min":0.49199999999999999,"15min":0.49199999999999999},"min":{"1min":0.38800000000000001,"5min":0.38800000000000001,"15min":0.38800000000000001},"max":{"1min":0.68500000000000005,"5min":0.68500000000000005,"15min":0.68500000000000005},"last":0.371}]},{"osd":2,"last update":"Sun Mar 24 11:15:31 2024","interfaces":[{"interface":"back","average":{"1min":0.48799999999999999,"5min":0.48799999999999999,"15min":0.48799999999999999},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.70499999999999996,"5min":0.70499999999999996,"15min":0.70499999999999996},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.52500000000000002,"5min":0.52500000000000002,"15min":0.52500000000000002},"min":{"1min":0.40300000000000002,"5min":0.40300000000000002,"15min":0.40300000000000002},"max":{"1min":0.71899999999999997,"5min":0.71899999999999997,"15min":0.71899999999999997},"last":0.39300000000000002}]},{"osd":4,"last update":"Sun Mar 24 11:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.374,"5min":0.374,"15min":0.374},"min":{"1min":0.27900000000000003,"5min":0.27900000000000003,"15min":0.27900000000000003},"max":{"1min":0.57399999999999995,"5min":0.57399999999999995,"15min":0.57399999999999995},"last":0.29199999999999998},{"interface":"front","average":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"min":{"1min":0.30199999999999999,"5min":0.30199999999999999,"15min":0.30199999999999999},"max":{"1min":0.58699999999999997,"5min":0.58699999999999997,"15min":0.58699999999999997},"last":0.27200000000000002}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.314}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297652,"kb_used_data":364,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93447500,"statfs":{"total":95995035648,"available":95690240000,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:16:04 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.35299999999999998,"15min":0.35599999999999998},"min":{"1min":0.216,"5min":0.216,"15min":0.216},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.33600000000000002,"5min":0.35799999999999998,"15min":0.36199999999999999},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.55200000000000005,"5min":0.55200000000000005,"15min":0.55200000000000005},"last":0.26400000000000001}]},{"osd":1,"last update":"Sun Mar 24 11:16:04 2024","interfaces":[{"interface":"back","average":{"1min":0.34000000000000002,"5min":0.372,"15min":0.378},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.34699999999999998,"5min":0.39100000000000001,"15min":0.39900000000000002},"min":{"1min":0.23000000000000001,"5min":0.23000000000000001,"15min":0.23000000000000001},"max":{"1min":0.48399999999999999,"5min":0.56499999999999995,"15min":0.56499999999999995},"last":0.30399999999999999}]},{"osd":3,"last update":"Sun Mar 24 11:15:28 2024","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.502,"15min":0.502},"min":{"1min":0.38600000000000001,"5min":0.38600000000000001,"15min":0.38600000000000001},"max":{"1min":0.59199999999999997,"5min":0.59199999999999997,"15min":0.59199999999999997},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.41699999999999998,"5min":0.41699999999999998,"15min":0.41699999999999998},"max":{"1min":0.66400000000000003,"5min":0.66400000000000003,"15min":0.66400000000000003},"last":0.33300000000000002}]},{"osd":4,"last update":"Sun Mar 24 11:15:53 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.39600000000000002,"5min":0.39600000000000002,"15min":0.39600000000000002},"max":{"1min":0.71999999999999997,"5min":0.71999999999999997,"15min":0.71999999999999997},"last":0.35699999999999998},{"interface":"front","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.44400000000000001,"5min":0.44400000000000001,"15min":0.44400000000000001},"max":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"last":0.41199999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.44400000000000001}]}]},{"osd":0,"up_from":9,"seq":38654705704,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sun Mar 24 11:15:39 2024","interfaces":[{"interface":"back","average":{"1min":0.36099999999999999,"5min":0.32100000000000001,"15min":0.314},"min":{"1min":0.21199999999999999,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.50900000000000001,"5min":0.50900000000000001,"15min":0.50900000000000001},"last":0.36599999999999999},{"interface":"front","average":{"1min":0.372,"5min":0.33500000000000002,"15min":0.32900000000000001},"min":{"1min":0.23100000000000001,"5min":0.17399999999999999,"15min":0.17399999999999999},"max":{"1min":0.55500000000000005,"5min":0.55500000000000005,"15min":0.55500000000000005},"last":0.314}]},{"osd":2,"last update":"Sun Mar 24 11:15:07 2024","interfaces":[{"interface":"back","average":{"1min":0.39400000000000002,"5min":0.39400000000000002,"15min":0.39400000000000002},"min":{"1min":0.23200000000000001,"5min":0.23200000000000001,"15min":0.23200000000000001},"max":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"last":0.33900000000000002},{"interface":"front","average":{"1min":0.41199999999999998,"5min":0.41199999999999998,"15min":0.41199999999999998},"min":{"1min":0.20799999999999999,"5min":0.20799999999999999,"15min":0.20799999999999999},"max":{"1min":0.623,"5min":0.623,"15min":0.623},"last":0.34999999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:15:29 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.35899999999999999,"5min":0.35899999999999999,"15min":0.35899999999999999},"max":{"1min":0.69099999999999995,"5min":0.69099999999999995,"15min":0.69099999999999995},"last":0.59199999999999997},{"interface":"front","average":{"1min":0.51700000000000002,"5min":0.51700000000000002,"15min":0.51700000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.70299999999999996,"5min":0.70299999999999996,"15min":0.70299999999999996},"last":0.55300000000000005}]},{"osd":4,"last update":"Sun Mar 24 11:15:51 2024","interfaces":[{"interface":"back","average":{"1min":0.55800000000000005,"5min":0.55800000000000005,"15min":0.55800000000000005},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.79700000000000004,"5min":0.79700000000000004,"15min":0.79700000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.57199999999999995,"5min":0.57199999999999995,"15min":0.57199999999999995},"min":{"1min":0.42099999999999999,"5min":0.42099999999999999,"15min":0.42099999999999999},"max":{"1min":0.72099999999999997,"5min":0.72099999999999997,"15min":0.72099999999999997},"last":0.60699999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.627}]}]},{"osd":1,"up_from":14,"seq":60129542179,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298232,"kb_used_data":944,"kb_used_omap":0,"kb_used_meta":297280,"kb_avail":93446920,"statfs":{"total":95995035648,"available":95689646080,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":304414720},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sun Mar 24 11:15:39 2024","interfaces":[{"interface":"back","average":{"1min":0.33700000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"min":{"1min":0.23999999999999999,"5min":0.21199999999999999,"15min":0.21199999999999999},"max":{"1min":0.58499999999999996,"5min":0.58499999999999996,"15min":0.58499999999999996},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.35899999999999999,"5min":0.34200000000000003,"15min":0.33900000000000002},"min":{"1min":0.24399999999999999,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"last":0.38400000000000001}]},{"osd":2,"last update":"Sun Mar 24 11:16:08 2024","interfaces":[{"interface":"back","average":{"1min":0.34300000000000003,"5min":0.38900000000000001,"15min":0.39700000000000002},"min":{"1min":0.26300000000000001,"5min":0.26300000000000001,"15min":0.26300000000000001},"max":{"1min":0.54500000000000004,"5min":0.57799999999999996,"15min":0.57799999999999996},"last":0.435},{"interface":"front","average":{"1min":0.32500000000000001,"5min":0.38800000000000001,"15min":0.39800000000000002},"min":{"1min":0.23300000000000001,"5min":0.23300000000000001,"15min":0.23300000000000001},"max":{"1min":0.53400000000000003,"5min":0.59899999999999998,"15min":0.59899999999999998},"last":0.41199999999999998}]},{"osd":3,"last update":"Sun Mar 24 11:15:29 2024","interfaces":[{"interface":"back","average":{"1min":0.52300000000000002,"5min":0.52300000000000002,"15min":0.52300000000000002},"min":{"1min":0.39000000000000001,"5min":0.39000000000000001,"15min":0.39000000000000001},"max":{"1min":0.76200000000000001,"5min":0.76200000000000001,"15min":0.76200000000000001},"last":0.58199999999999996},{"interface":"front","average":{"1min":0.52600000000000002,"5min":0.52600000000000002,"15min":0.52600000000000002},"min":{"1min":0.39800000000000002,"5min":0.39800000000000002,"15min":0.39800000000000002},"max":{"1min":0.746,"5min":0.746,"15min":0.746},"last":0.49399999999999999}]},{"osd":4,"last update":"Sun Mar 24 11:15:52 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.41499999999999998,"5min":0.41499999999999998,"15min":0.41499999999999998},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.52800000000000002,"5min":0.52800000000000002,"15min":0.52800000000000002},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.84599999999999997,"5min":0.84599999999999997,"15min":0.84599999999999997},"last":0.54100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48399999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-03-24T11:16:13.811 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-03-24T11:16:13.811 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-03-24T11:16:13.811 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-03-24T11:16:13.812 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph health --format=json 2024-03-24T11:16:14.065 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:14.478 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:14 smithi100 ceph-mon[28941]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:14.479 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:14 smithi100 ceph-mon[28941]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:16:14.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:14 smithi105 ceph-mon[36437]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:14.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:14 smithi105 ceph-mon[36437]: from='client.14432 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-03-24T11:16:15.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 -- 172.21.15.100:0/1061391664 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800101d70 msgr2=0x7f9800102140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 --2- 172.21.15.100:0/1061391664 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800101d70 0x7f9800102140 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f97f8009ce0 tx=0x7f97f8004d10 comp rx=0 tx=0).stop 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 -- 172.21.15.100:0/1061391664 shutdown_connections 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 --2- 172.21.15.100:0/1061391664 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9800102710 0x7f98000fbd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 --2- 172.21.15.100:0/1061391664 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800101d70 0x7f9800102140 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 -- 172.21.15.100:0/1061391664 >> 172.21.15.100:0/1061391664 conn(0x7f9800072bc0 msgr2=0x7f9800072fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 -- 172.21.15.100:0/1061391664 shutdown_connections 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.730+0000 7f9806a8e700 1 -- 172.21.15.100:0/1061391664 wait complete. 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f9806a8e700 1 Processor -- start 2024-03-24T11:16:15.734 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f9806a8e700 1 -- start start 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f9806a8e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9800101d70 0x7f980010edd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f9806a8e700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 0x7f980010f310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f9806a8e700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980010f850 con 0x7f9800102710 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f9806a8e700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f980010f990 con 0x7f9800101d70 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f97fffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 0x7f980010f310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f97fffff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 0x7f980010f310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:54014/0 (socket says 172.21.15.100:54014) 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f97fffff700 1 -- 172.21.15.100:0/2540138199 learned_addr learned my addr 172.21.15.100:0/2540138199 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f980482a700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9800101d70 0x7f980010edd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f97fffff700 1 -- 172.21.15.100:0/2540138199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9800101d70 msgr2=0x7f980010edd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f97fffff700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9800101d70 0x7f980010edd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:15.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.731+0000 7f97fffff700 1 -- 172.21.15.100:0/2540138199 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f97f8009940 con 0x7f9800102710 2024-03-24T11:16:15.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.732+0000 7f97fffff700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 0x7f980010f310 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97f000b790 tx=0x7f97f0009220 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:15.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.732+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97f000f6f0 con 0x7f9800102710 2024-03-24T11:16:15.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.732+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f97f000fd20 con 0x7f9800102710 2024-03-24T11:16:15.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.732+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9800117ce0 con 0x7f9800102710 2024-03-24T11:16:15.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.732+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f97f00103a0 con 0x7f9800102710 2024-03-24T11:16:15.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.732+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9800118240 con 0x7f9800102710 2024-03-24T11:16:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.733+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98000ff080 con 0x7f9800102710 2024-03-24T11:16:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.733+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f97f0011400 con 0x7f9800102710 2024-03-24T11:16:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.734+0000 7f97fd7fa700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f97ec0692a0 0x7f97ec06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.734+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f97f0086c80 con 0x7f9800102710 2024-03-24T11:16:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.734+0000 7f980482a700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f97ec0692a0 0x7f97ec06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:15.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.734+0000 7f980482a700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f97ec0692a0 0x7f97ec06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f97f800b740 tx=0x7f97f8004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:15.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.738+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f97f00532f0 con 0x7f9800102710 2024-03-24T11:16:15.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.966+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9800108ee0 con 0x7f9800102710 2024-03-24T11:16:15.967 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.966+0000 7f97fd7fa700 1 -- 172.21.15.100:0/2540138199 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+46 (secure 0 0 0) 0x7f97f0056910 con 0x7f9800102710 2024-03-24T11:16:15.968 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:15.968 INFO:teuthology.orchestra.run.smithi100.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f97ec0692a0 msgr2=0x7f97ec06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f97ec0692a0 0x7f97ec06b750 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7f97f800b740 tx=0x7f97f8004d10 comp rx=0 tx=0).stop 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 msgr2=0x7f980010f310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 0x7f980010f310 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7f97f000b790 tx=0x7f97f0009220 comp rx=0 tx=0).stop 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 shutdown_connections 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f97ec0692a0 0x7f97ec06b750 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9800101d70 0x7f980010edd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 --2- 172.21.15.100:0/2540138199 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f9800102710 0x7f980010f310 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 >> 172.21.15.100:0/2540138199 conn(0x7f9800072bc0 msgr2=0x7f98000fae90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 shutdown_connections 2024-03-24T11:16:15.970 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:15.968+0000 7f9806a8e700 1 -- 172.21.15.100:0/2540138199 wait complete. 2024-03-24T11:16:16.639 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:16 smithi100 ceph-mon[28941]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:16.639 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:16 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2540138199' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:16:16.663 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy done 2024-03-24T11:16:16.664 INFO:tasks.cephadm:Setup complete, yielding 2024-03-24T11:16:16.664 INFO:teuthology.run_tasks:Running task print... 2024-03-24T11:16:16.674 INFO:teuthology.task.print:**** done end installing quincy cephadm ... 2024-03-24T11:16:16.674 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:16.682 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:16.683 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph config set mgr mgr/cephadm/use_repo_digest true --force' 2024-03-24T11:16:16.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:16 smithi105 ceph-mon[36437]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:16.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:16 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2540138199' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-03-24T11:16:16.939 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:18.477 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:18 smithi100 ceph-mon[28941]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:18.690 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 -- 172.21.15.100:0/3602668685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 msgr2=0x7f86a40fd280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 --2- 172.21.15.100:0/3602668685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a40fd280 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7f8694004940 tx=0x7f8694009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 -- 172.21.15.100:0/3602668685 shutdown_connections 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 --2- 172.21.15.100:0/3602668685 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86a40fdfe0 0x7f86a40fe450 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 --2- 172.21.15.100:0/3602668685 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a40fd280 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 -- 172.21.15.100:0/3602668685 >> 172.21.15.100:0/3602668685 conn(0x7f86a40f8550 msgr2=0x7f86a40fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 -- 172.21.15.100:0/3602668685 shutdown_connections 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.686+0000 7f86a939f700 1 -- 172.21.15.100:0/3602668685 wait complete. 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a939f700 1 Processor -- start 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a939f700 1 -- start start 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a939f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a410d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a939f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86a40fdfe0 0x7f86a410de70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a939f700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a410e470 con 0x7f86a40fce70 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a939f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86a410e5e0 con 0x7f86a40fdfe0 2024-03-24T11:16:18.691 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a410d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a410d930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:54288/0 (socket says 172.21.15.100:54288) 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 -- 172.21.15.100:0/337803129 learned_addr learned my addr 172.21.15.100:0/337803129 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f869a7fc700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86a40fdfe0 0x7f86a410de70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 -- 172.21.15.100:0/337803129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86a40fdfe0 msgr2=0x7f86a410de70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86a40fdfe0 0x7f86a410de70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 -- 172.21.15.100:0/337803129 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8694005040 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.687+0000 7f86a2ffd700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a410d930 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f869400b3b0 tx=0x7f8694009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.688+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869400e040 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.688+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86a4112fe0 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.688+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f869400bbf0 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.688+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f869401c630 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.688+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86a4113510 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.689+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86a404c710 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.689+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f869400bd60 con 0x7f86a40fce70 2024-03-24T11:16:18.692 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.690+0000 7f869bfff700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f86900692a0 0x7f869006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:18.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.690+0000 7f869a7fc700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f86900692a0 0x7f869006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:18.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.690+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8694087b00 con 0x7f86a40fce70 2024-03-24T11:16:18.693 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.690+0000 7f869a7fc700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f86900692a0 0x7f869006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f868c0041d0 tx=0x7f868c0043c0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:18.694 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.694+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8694057c00 con 0x7f86a40fce70 2024-03-24T11:16:18.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:18 smithi105 ceph-mon[36437]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:18.878 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.877+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/use_repo_digest}] v 0) v1 -- 0x7f86a41058d0 con 0x7f86a40fce70 2024-03-24T11:16:18.882 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.882+0000 7f869bfff700 1 -- 172.21.15.100:0/337803129 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/use_repo_digest}]=0 v20)=0 v20) v1 ==== 143+0+0 (secure 0 0 0) 0x7f8694057790 con 0x7f86a40fce70 2024-03-24T11:16:18.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f86900692a0 msgr2=0x7f869006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:18.884 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f86900692a0 0x7f869006b750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7f868c0041d0 tx=0x7f868c0043c0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 msgr2=0x7f86a410d930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a410d930 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7f869400b3b0 tx=0x7f8694009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 shutdown_connections 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f86900692a0 0x7f869006b750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f86a40fce70 0x7f86a410d930 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 --2- 172.21.15.100:0/337803129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f86a40fdfe0 0x7f86a410de70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 >> 172.21.15.100:0/337803129 conn(0x7f86a40f8550 msgr2=0x7f86a40f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 shutdown_connections 2024-03-24T11:16:18.885 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:18.884+0000 7f86a939f700 1 -- 172.21.15.100:0/337803129 wait complete. 2024-03-24T11:16:19.706 INFO:teuthology.run_tasks:Running task print... 2024-03-24T11:16:19.716 INFO:teuthology.task.print:**** done cephadm.shell ceph config set mgr... 2024-03-24T11:16:19.716 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:19.732 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:19.732 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph orch status' 2024-03-24T11:16:20.005 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:20.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:19 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/337803129' entity='client.admin' 2024-03-24T11:16:20.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:20.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:19 smithi100 ceph-mon[28941]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:20.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:20.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:20.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:19 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:19 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/337803129' entity='client.admin' 2024-03-24T11:16:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:19 smithi105 ceph-mon[36437]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:20.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:20.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:19 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- 172.21.15.100:0/3540755195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 msgr2=0x7fd2d40fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/3540755195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d40fc0e0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fd2d0004940 tx=0x7fd2d0009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- 172.21.15.100:0/3540755195 shutdown_connections 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/3540755195 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 0x7fd2d40fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/3540755195 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d40fc0e0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- 172.21.15.100:0/3540755195 >> 172.21.15.100:0/3540755195 conn(0x7fd2d40f7430 msgr2=0x7fd2d40f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- 172.21.15.100:0/3540755195 shutdown_connections 2024-03-24T11:16:21.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- 172.21.15.100:0/3540755195 wait complete. 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 Processor -- start 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- start start 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d410da00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 0x7fd2d410df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d410e540 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.609+0000 7fd2db2c3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2d410e6b0 con 0x7fd2d40fcf80 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d410da00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d410da00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:54304/0 (socket says 172.21.15.100:54304) 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 -- 172.21.15.100:0/1952688451 learned_addr learned my addr 172.21.15.100:0/1952688451 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d885e700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 0x7fd2d410df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 -- 172.21.15.100:0/1952688451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 msgr2=0x7fd2d410df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 0x7fd2d410df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 -- 172.21.15.100:0/1952688451 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2d0005040 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d885e700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 0x7fd2d410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:16:21.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2d905f700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d410da00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000b3b0 tx=0x7fd2d0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2d000e040 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2d4113010 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd2d000bcc0 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd2d001c630 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.610+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2d41135d0 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.611+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2d4101b10 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.612+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd2d0013980 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.612+0000 7fd2c9ffb700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd2c00692a0 0x7fd2c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.612+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd2d0087c80 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.612+0000 7fd2d885e700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd2c00692a0 0x7fd2c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:21.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.613+0000 7fd2d885e700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd2c00692a0 0x7fd2c006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4007fd0 tx=0x7fd2c4009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:21.617 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.616+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd2d0057d00 con 0x7fd2d40fbcd0 2024-03-24T11:16:21.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.817+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd2d405e400 con 0x7fd2c00692a0 2024-03-24T11:16:21.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.819+0000 7fd2c9ffb700 1 -- 172.21.15.100:0/1952688451 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+43 (secure 0 0 0) 0x7fd2d405e400 con 0x7fd2c00692a0 2024-03-24T11:16:21.819 INFO:teuthology.orchestra.run.smithi100.stdout:Backend: cephadm 2024-03-24T11:16:21.819 INFO:teuthology.orchestra.run.smithi100.stdout:Available: Yes 2024-03-24T11:16:21.819 INFO:teuthology.orchestra.run.smithi100.stdout:Paused: No 2024-03-24T11:16:21.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.820+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd2c00692a0 msgr2=0x7fd2c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:21.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.820+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd2c00692a0 0x7fd2c006b750 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7fd2c4007fd0 tx=0x7fd2c4009040 comp rx=0 tx=0).stop 2024-03-24T11:16:21.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 msgr2=0x7fd2d410da00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:21.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d410da00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7fd2d000b3b0 tx=0x7fd2d0009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 shutdown_connections 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd2c00692a0 0x7fd2c006b750 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd2d40fbcd0 0x7fd2d410da00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 --2- 172.21.15.100:0/1952688451 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2d40fcf80 0x7fd2d410df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 >> 172.21.15.100:0/1952688451 conn(0x7fd2d40f7430 msgr2=0x7fd2d4066860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 shutdown_connections 2024-03-24T11:16:21.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:21.821+0000 7fd2db2c3700 1 -- 172.21.15.100:0/1952688451 wait complete. 2024-03-24T11:16:22.491 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:22 smithi100 ceph-mon[28941]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:22.516 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ps' 2024-03-24T11:16:22.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:22 smithi105 ceph-mon[36437]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:22.770 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:23.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:23 smithi105 ceph-mon[36437]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:23.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:23.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:23 smithi100 ceph-mon[28941]: from='client.14444 -' entity='client.admin' cmd=[{"prefix": "orch status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:23.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:24.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 -- 172.21.15.100:0/3756129637 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80fe450 msgr2=0x7f12a80fe860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:24.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/3756129637 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80fe450 0x7f12a80fe860 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f1298004940 tx=0x7f1298009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:24.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 -- 172.21.15.100:0/3756129637 shutdown_connections 2024-03-24T11:16:24.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/3756129637 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80feda0 0x7f12a80faee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:24.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/3756129637 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80fe450 0x7f12a80fe860 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 -- 172.21.15.100:0/3756129637 >> 172.21.15.100:0/3756129637 conn(0x7f12a80f6c10 msgr2=0x7f12a80f9020 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 -- 172.21.15.100:0/3756129637 shutdown_connections 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.397+0000 7f12ad1ce700 1 -- 172.21.15.100:0/3756129637 wait complete. 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12ad1ce700 1 Processor -- start 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12ad1ce700 1 -- start start 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12ad1ce700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80fe450 0x7f12a810d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12ad1ce700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 0x7f12a810de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12ad1ce700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a810e4e0 con 0x7f12a80feda0 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12ad1ce700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a8112e80 con 0x7f12a80fe450 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12a659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 0x7f12a810de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12a659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 0x7f12a810de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:54312/0 (socket says 172.21.15.100:54312) 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.398+0000 7f12a659c700 1 -- 172.21.15.100:0/654729617 learned_addr learned my addr 172.21.15.100:0/654729617 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:24.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12a6d9d700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80fe450 0x7f12a810d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12a659c700 1 -- 172.21.15.100:0/654729617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80fe450 msgr2=0x7f12a810d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12a659c700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80fe450 0x7f12a810d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12a659c700 1 -- 172.21.15.100:0/654729617 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1298005040 con 0x7f12a80feda0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12a6d9d700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80fe450 0x7f12a810d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12a659c700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 0x7f12a810de50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f129c00b790 tx=0x7f129c009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f129c00f770 con 0x7f12a80feda0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f129c00fda0 con 0x7f12a80feda0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a81130e0 con 0x7f12a80feda0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f129c00d310 con 0x7f12a80feda0 2024-03-24T11:16:24.402 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.399+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a8113650 con 0x7f12a80feda0 2024-03-24T11:16:24.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.400+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a804c710 con 0x7f12a80feda0 2024-03-24T11:16:24.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.400+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f129c010420 con 0x7f12a80feda0 2024-03-24T11:16:24.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.401+0000 7f128f7fe700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f12900692a0 0x7f129006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:24.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.401+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f129c086e60 con 0x7f12a80feda0 2024-03-24T11:16:24.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.401+0000 7f12a6d9d700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f12900692a0 0x7f129006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:24.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.401+0000 7f12a6d9d700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f12900692a0 0x7f129006b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:24.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.405+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f129c056f60 con 0x7f12a80feda0 2024-03-24T11:16:24.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:24 smithi105 ceph-mon[36437]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:24.600 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.600+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f12a81024d0 con 0x7f12900692a0 2024-03-24T11:16:24.600 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:24 smithi100 ceph-mon[28941]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.609+0000 7f128f7fe700 1 -- 172.21.15.100:0/654729617 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+2682 (secure 0 0 0) 0x7f12a81024d0 con 0x7f12900692a0 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.smithi100 smithi100 *:9093,9094 running (3m) 2m ago 5m 26.2M - 0.25.0 c8568f914cd2 1c0d9b7ea6a4 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:crash.smithi100 smithi100 running (5m) 2m ago 5m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 6221438d109b 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:crash.smithi105 smithi105 running (4m) 68s ago 4m 6656k - 17.2.7-726-g9c4e0c34 837124da804f 6007c22dfa6c 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.smithi100 smithi100 *:3000 running (3m) 2m ago 4m 77.8M - 9.4.7 954c08fa6188 884de3e3cb33 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.smithi100.smxgyi smithi100 *:9283 running (6m) 2m ago 6m 472M - 17.2.7-726-g9c4e0c34 837124da804f 3a17926ffe53 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.smithi105.nmspmy smithi105 *:8443,9283 running (4m) 68s ago 4m 432M - 17.2.7-726-g9c4e0c34 837124da804f 20760d28e4ea 2024-03-24T11:16:24.610 INFO:teuthology.orchestra.run.smithi100.stdout:mon.smithi100 smithi100 running (6m) 2m ago 6m 40.2M 2048M 17.2.7-726-g9c4e0c34 837124da804f f7fbf127ce8a 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:mon.smithi105 smithi105 running (4m) 68s ago 4m 38.9M 2048M 17.2.7-726-g9c4e0c34 837124da804f 243836c787a4 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.smithi100 smithi100 *:9100 running (5m) 2m ago 5m 15.8M - 1.5.0 0da6a335fe13 71e6d8a3aef2 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.smithi105 smithi105 *:9100 running (4m) 68s ago 4m 16.1M - 1.5.0 0da6a335fe13 8fd41687715a 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (3m) 2m ago 3m 39.0M 4288M 17.2.7-726-g9c4e0c34 837124da804f d8dbc515015a 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (2m) 2m ago 2m 42.3M 4288M 17.2.7-726-g9c4e0c34 837124da804f 8c2b1a04b4bf 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (2m) 2m ago 2m 38.4M 4288M 17.2.7-726-g9c4e0c34 837124da804f e04c18a8a009 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi105 running (2m) 68s ago 2m 42.2M 5312M 17.2.7-726-g9c4e0c34 837124da804f 97b4dd2a4b6e 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi105 running (99s) 68s ago 99s 40.0M 5312M 17.2.7-726-g9c4e0c34 837124da804f 1f71a14b0353 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi105 running (73s) 68s ago 73s 11.0M 5312M 17.2.7-726-g9c4e0c34 837124da804f 211e50c6219d 2024-03-24T11:16:24.611 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.smithi100 smithi100 *:9095 running (3m) 2m ago 4m 37.0M - 2.43.0 a07b618ecd1d 6a8a1455db93 2024-03-24T11:16:24.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f12900692a0 msgr2=0x7f129006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:24.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f12900692a0 0x7f129006b750 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f1298000c00 tx=0x7f1298009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 msgr2=0x7f12a810de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 0x7f12a810de50 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f129c00b790 tx=0x7f129c009250 comp rx=0 tx=0).stop 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 shutdown_connections 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f12900692a0 0x7f129006b750 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12a80fe450 0x7f12a810d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 --2- 172.21.15.100:0/654729617 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12a80feda0 0x7f12a810de50 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.611+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 >> 172.21.15.100:0/654729617 conn(0x7f12a80f6c10 msgr2=0x7f12a80f8230 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.612+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 shutdown_connections 2024-03-24T11:16:24.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:24.612+0000 7f12ad1ce700 1 -- 172.21.15.100:0/654729617 wait complete. 2024-03-24T11:16:25.224 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph orch ls' 2024-03-24T11:16:25.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:25 smithi105 ceph-mon[36437]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:25.479 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:25.601 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:25 smithi100 ceph-mon[28941]: from='client.14448 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:26.552 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:26 smithi100 ceph-mon[28941]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:26.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:26 smithi105 ceph-mon[36437]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 -- 172.21.15.100:0/4087083656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 msgr2=0x7f5fe0102320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 --2- 172.21.15.100:0/4087083656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe0102320 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0004940 tx=0x7f5fd0009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 -- 172.21.15.100:0/4087083656 shutdown_connections 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 --2- 172.21.15.100:0/4087083656 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe0102320 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 --2- 172.21.15.100:0/4087083656 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 0x7f5fe00ff9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 -- 172.21.15.100:0/4087083656 >> 172.21.15.100:0/4087083656 conn(0x7f5fe00f7430 msgr2=0x7f5fe00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:27.080 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 -- 172.21.15.100:0/4087083656 shutdown_connections 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 -- 172.21.15.100:0/4087083656 wait complete. 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 Processor -- start 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.078+0000 7f5fe6177700 1 -- start start 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fe6177700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 0x7f5fe010dac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fe6177700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe010e000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fe6177700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe010e690 con 0x7f5fe00fff00 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fe6177700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fe010ee30 con 0x7f5fe00662e0 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe010e000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe010e000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:54334/0 (socket says 172.21.15.100:54334) 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 -- 172.21.15.100:0/994539164 learned_addr learned my addr 172.21.15.100:0/994539164 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdf7fe700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 0x7f5fe010dac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:27.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 -- 172.21.15.100:0/994539164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 msgr2=0x7f5fe010dac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 0x7f5fe010dac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 -- 172.21.15.100:0/994539164 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fd0005040 con 0x7f5fe00fff00 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdf7fe700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 0x7f5fe010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fdeffd700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe010e000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009150 tx=0x7f5fd0009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd000e040 con 0x7f5fe00fff00 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5fd000bbf0 con 0x7f5fe00fff00 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fe010f0b0 con 0x7f5fe00fff00 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5fd001c630 con 0x7f5fe00fff00 2024-03-24T11:16:27.082 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.079+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fe010f670 con 0x7f5fe00fff00 2024-03-24T11:16:27.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.080+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fe005f5f0 con 0x7f5fe00fff00 2024-03-24T11:16:27.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.081+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5fd000bd60 con 0x7f5fe00fff00 2024-03-24T11:16:27.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.081+0000 7f5fc7fff700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5fc80692b0 0x7f5fc806b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:27.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.081+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5fd00882a0 con 0x7f5fe00fff00 2024-03-24T11:16:27.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.081+0000 7f5fdf7fe700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5fc80692b0 0x7f5fc806b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:27.083 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.082+0000 7f5fdf7fe700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5fc80692b0 0x7f5fc806b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0066020 tx=0x7f5fd4008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:27.086 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.086+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5fd0058450 con 0x7f5fe00fff00 2024-03-24T11:16:27.285 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.284+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch ls", "target": ["mon-mgr", ""]}) v1 -- 0x7f5fe010f950 con 0x7f5fc80692b0 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.288+0000 7f5fc7fff700 1 -- 172.21.15.100:0/994539164 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+1179 (secure 0 0 0) 0x7f5fe010f950 con 0x7f5fc80692b0 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:NAME PORTS RUNNING REFRESHED AGE PLACEMENT 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager ?:9093,9094 1/1 2m ago 5m count:1 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:crash 2/2 2m ago 5m * 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:grafana ?:3000 1/1 2m ago 5m count:1 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:mgr 2/2 2m ago 5m count:2 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:mon 2/2 2m ago 5m smithi100:172.21.15.100=smithi100;smithi105:172.21.15.105=smithi105;count:2 2024-03-24T11:16:27.289 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter ?:9100 2/2 2m ago 5m * 2024-03-24T11:16:27.290 INFO:teuthology.orchestra.run.smithi100.stdout:osd 6 2m ago - 2024-03-24T11:16:27.290 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus ?:9095 1/1 2m ago 5m count:1 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5fc80692b0 msgr2=0x7f5fc806b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5fc80692b0 0x7f5fc806b760 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f5fe0066020 tx=0x7f5fd4008040 comp rx=0 tx=0).stop 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 msgr2=0x7f5fe010e000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe010e000 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f5fd0009150 tx=0x7f5fd0009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 shutdown_connections 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5fc80692b0 0x7f5fc806b760 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5fe00662e0 0x7f5fe010dac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 --2- 172.21.15.100:0/994539164 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5fe00fff00 0x7f5fe010e000 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:27.291 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.290+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 >> 172.21.15.100:0/994539164 conn(0x7f5fe00f7430 msgr2=0x7f5fe00f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:27.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.291+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 shutdown_connections 2024-03-24T11:16:27.292 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:27.291+0000 7f5fe6177700 1 -- 172.21.15.100:0/994539164 wait complete. 2024-03-24T11:16:27.967 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph orch host ls' 2024-03-24T11:16:28.220 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:28.545 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:28 smithi100 ceph-mon[28941]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:28.545 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:28 smithi100 ceph-mon[28941]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:28.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:28 smithi105 ceph-mon[36437]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:28.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:28 smithi105 ceph-mon[36437]: from='client.14452 -' entity='client.admin' cmd=[{"prefix": "orch ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:29.827 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 -- 172.21.15.100:0/3057803587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fcf70 msgr2=0x7fced40fd3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 --2- 172.21.15.100:0/3057803587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fcf70 0x7fced40fd3e0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7fcebc004940 tx=0x7fcebc009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 -- 172.21.15.100:0/3057803587 shutdown_connections 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 --2- 172.21.15.100:0/3057803587 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fcf70 0x7fced40fd3e0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 --2- 172.21.15.100:0/3057803587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fced40fbe00 0x7fced40fc210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 -- 172.21.15.100:0/3057803587 >> 172.21.15.100:0/3057803587 conn(0x7fced40f74e0 msgr2=0x7fced40f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 -- 172.21.15.100:0/3057803587 shutdown_connections 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.825+0000 7fced9500700 1 -- 172.21.15.100:0/3057803587 wait complete. 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced9500700 1 Processor -- start 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced9500700 1 -- start start 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced9500700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 0x7fced4110770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced9500700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fced40fcf70 0x7fced4112cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:29.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced9500700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4066c90 con 0x7fced40fbe00 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced9500700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fced4066dd0 con 0x7fced40fcf70 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 0x7fced4110770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 0x7fced4110770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47028/0 (socket says 172.21.15.100:47028) 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 -- 172.21.15.100:0/574106715 learned_addr learned my addr 172.21.15.100:0/574106715 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced27fc700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fced40fcf70 0x7fced4112cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 -- 172.21.15.100:0/574106715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fced40fcf70 msgr2=0x7fced4112cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fced40fcf70 0x7fced4112cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 -- 172.21.15.100:0/574106715 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcebc005040 con 0x7fced40fbe00 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.826+0000 7fced2ffd700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 0x7fced4110770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcec400bbe0 tx=0x7fcec4009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.827+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcec40092e0 con 0x7fced40fbe00 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.827+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fcec400cb20 con 0x7fced40fbe00 2024-03-24T11:16:29.829 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.827+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fced4113320 con 0x7fced40fbe00 2024-03-24T11:16:29.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.827+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcec4015db0 con 0x7fced40fbe00 2024-03-24T11:16:29.830 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.827+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fced41138e0 con 0x7fced40fbe00 2024-03-24T11:16:29.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.828+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fced405f5f0 con 0x7fced40fbe00 2024-03-24T11:16:29.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.828+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fcec4012070 con 0x7fced40fbe00 2024-03-24T11:16:29.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.828+0000 7fcecb7fe700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcec00692a0 0x7fcec006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:29.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.828+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcec4086a00 con 0x7fced40fbe00 2024-03-24T11:16:29.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.828+0000 7fced27fc700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcec00692a0 0x7fcec006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:29.831 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.829+0000 7fced27fc700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcec00692a0 0x7fcec006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00b970 tx=0x7fcebc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:29.833 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:29.833+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fcec4052ff0 con 0x7fced40fbe00 2024-03-24T11:16:30.026 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.025+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""]}) v1 -- 0x7fced4101210 con 0x7fcec00692a0 2024-03-24T11:16:30.027 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.027+0000 7fcecb7fe700 1 -- 172.21.15.100:0/574106715 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+148 (secure 0 0 0) 0x7fced4101210 con 0x7fcec00692a0 2024-03-24T11:16:30.027 INFO:teuthology.orchestra.run.smithi100.stdout:HOST ADDR LABELS STATUS 2024-03-24T11:16:30.027 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100 172.21.15.100 2024-03-24T11:16:30.027 INFO:teuthology.orchestra.run.smithi100.stdout:smithi105 172.21.15.105 2024-03-24T11:16:30.027 INFO:teuthology.orchestra.run.smithi100.stdout:2 hosts in cluster 2024-03-24T11:16:30.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.028+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcec00692a0 msgr2=0x7fcec006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:30.029 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.028+0000 7fced9500700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcec00692a0 0x7fcec006b750 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fcebc00b970 tx=0x7fcebc009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.028+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 msgr2=0x7fced4110770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 0x7fced4110770 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fcec400bbe0 tx=0x7fcec4009840 comp rx=0 tx=0).stop 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 shutdown_connections 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fcec00692a0 0x7fcec006b750 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fced40fbe00 0x7fced4110770 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 --2- 172.21.15.100:0/574106715 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fced40fcf70 0x7fced4112cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 >> 172.21.15.100:0/574106715 conn(0x7fced40f74e0 msgr2=0x7fced40f8b40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 shutdown_connections 2024-03-24T11:16:30.030 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:30.029+0000 7fced9500700 1 -- 172.21.15.100:0/574106715 wait complete. 2024-03-24T11:16:30.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:30 smithi105 ceph-mon[36437]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:30.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:30 smithi100 ceph-mon[28941]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:30.698 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph orch device ls' 2024-03-24T11:16:30.954 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:31.328 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:31 smithi100 ceph-mon[28941]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:31.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:31 smithi105 ceph-mon[36437]: from='client.14456 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:32.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:32 smithi100 ceph-mon[28941]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:32.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:32 smithi105 ceph-mon[36437]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:32.607 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.605+0000 7faf8f494700 1 -- 172.21.15.100:0/561268906 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fcf80 msgr2=0x7faf880fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.605+0000 7faf8f494700 1 --2- 172.21.15.100:0/561268906 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fcf80 0x7faf880fd3f0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7faf78004940 tx=0x7faf78009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- 172.21.15.100:0/561268906 shutdown_connections 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 --2- 172.21.15.100:0/561268906 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fcf80 0x7faf880fd3f0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 --2- 172.21.15.100:0/561268906 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf880fbcd0 0x7faf880fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- 172.21.15.100:0/561268906 >> 172.21.15.100:0/561268906 conn(0x7faf880f7450 msgr2=0x7faf880f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- 172.21.15.100:0/561268906 shutdown_connections 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- 172.21.15.100:0/561268906 wait complete. 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 Processor -- start 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- start start 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 0x7faf88066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf880fcf80 0x7faf88112720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88112c60 con 0x7faf880fbcd0 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8f494700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf88112da0 con 0x7faf880fcf80 2024-03-24T11:16:32.608 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.606+0000 7faf8d230700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 0x7faf88066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8d230700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 0x7faf88066cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47048/0 (socket says 172.21.15.100:47048) 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8d230700 1 -- 172.21.15.100:0/48429814 learned_addr learned my addr 172.21.15.100:0/48429814 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8ca2f700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf880fcf80 0x7faf88112720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8d230700 1 -- 172.21.15.100:0/48429814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf880fcf80 msgr2=0x7faf88112720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8d230700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf880fcf80 0x7faf88112720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8d230700 1 -- 172.21.15.100:0/48429814 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf78005040 con 0x7faf880fbcd0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8d230700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 0x7faf88066cd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faf8400c8c0 tx=0x7faf840044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf8400a980 con 0x7faf880fbcd0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf8400ce60 con 0x7faf880fbcd0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf88112fa0 con 0x7faf880fbcd0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf840115e0 con 0x7faf880fbcd0 2024-03-24T11:16:32.609 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.607+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf88113530 con 0x7faf880fbcd0 2024-03-24T11:16:32.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.608+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf8804c710 con 0x7faf880fbcd0 2024-03-24T11:16:32.610 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.609+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7faf84012400 con 0x7faf880fbcd0 2024-03-24T11:16:32.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.609+0000 7faf7dffb700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf740692a0 0x7faf7406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:32.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.609+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7faf84086e30 con 0x7faf880fbcd0 2024-03-24T11:16:32.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.609+0000 7faf8ca2f700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf740692a0 0x7faf7406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:32.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.609+0000 7faf8ca2f700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf740692a0 0x7faf7406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b3b0 tx=0x7faf78009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:32.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.613+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7faf84056fd0 con 0x7faf880fbcd0 2024-03-24T11:16:32.806 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.806+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch device ls", "target": ["mon-mgr", ""]}) v1 -- 0x7faf881137d0 con 0x7faf740692a0 2024-03-24T11:16:32.808 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.807+0000 7faf7dffb700 1 -- 172.21.15.100:0/48429814 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+399 (secure 0 0 0) 0x7faf881137d0 con 0x7faf740692a0 2024-03-24T11:16:32.808 INFO:teuthology.orchestra.run.smithi100.stdout:HOST PATH TYPE DEVICE ID SIZE AVAILABLE REFRESHED REJECT REASONS 2024-03-24T11:16:32.808 INFO:teuthology.orchestra.run.smithi100.stdout:smithi100 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400WR400BGN 372G No 2m ago Has a FileSystem, LVM detected 2024-03-24T11:16:32.808 INFO:teuthology.orchestra.run.smithi100.stdout:smithi105 /dev/nvme0n1 ssd INTEL_SSDPEDMD400G4_PHFT620400TR400BGN 372G No 73s ago Has a FileSystem, LVM detected 2024-03-24T11:16:32.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf740692a0 msgr2=0x7faf7406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:32.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf740692a0 0x7faf7406b750 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7faf7800b3b0 tx=0x7faf78009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:32.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 msgr2=0x7faf88066cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:32.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 0x7faf88066cd0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7faf8400c8c0 tx=0x7faf840044c0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.810 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 shutdown_connections 2024-03-24T11:16:32.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf740692a0 0x7faf7406b750 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf880fbcd0 0x7faf88066cd0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 --2- 172.21.15.100:0/48429814 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf880fcf80 0x7faf88112720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:32.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.809+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 >> 172.21.15.100:0/48429814 conn(0x7faf880f7450 msgr2=0x7faf880f8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:32.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.810+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 shutdown_connections 2024-03-24T11:16:32.811 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:32.810+0000 7faf8f494700 1 -- 172.21.15.100:0/48429814 wait complete. 2024-03-24T11:16:33.495 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:33.505 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:33.505 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph fs volume create cephfs --placement=4' 2024-03-24T11:16:33.762 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:34.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:34 smithi105 ceph-mon[36437]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:34.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:34 smithi105 ceph-mon[36437]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:34.567 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:34 smithi100 ceph-mon[28941]: from='client.14460 -' entity='client.admin' cmd=[{"prefix": "orch device ls", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:34.567 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:34 smithi100 ceph-mon[28941]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:35.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- 172.21.15.100:0/2711907457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 msgr2=0x7f70400fd420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:35.382 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 --2- 172.21.15.100:0/2711907457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f70400fd420 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7f7030004940 tx=0x7f7030009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- 172.21.15.100:0/2711907457 shutdown_connections 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 --2- 172.21.15.100:0/2711907457 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f70400fd420 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 --2- 172.21.15.100:0/2711907457 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70400fbd00 0x7f70400fc110 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- 172.21.15.100:0/2711907457 >> 172.21.15.100:0/2711907457 conn(0x7f70400f7480 msgr2=0x7f70400f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- 172.21.15.100:0/2711907457 shutdown_connections 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- 172.21.15.100:0/2711907457 wait complete. 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 Processor -- start 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- start start 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70400fbd00 0x7f7040066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f7040112730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040112c70 con 0x7f70400fcfb0 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.381+0000 7f7048147700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7040112db0 con 0x7f70400fbd00 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f7040112730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f7040112730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:47072/0 (socket says 172.21.15.100:47072) 2024-03-24T11:16:35.383 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 -- 172.21.15.100:0/640782440 learned_addr learned my addr 172.21.15.100:0/640782440 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f7045ee3700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70400fbd00 0x7f7040066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 -- 172.21.15.100:0/640782440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70400fbd00 msgr2=0x7f7040066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70400fbd00 0x7f7040066cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 -- 172.21.15.100:0/640782440 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7030005040 con 0x7f70400fcfb0 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70456e2700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f7040112730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7030009150 tx=0x7f7030009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f703000e040 con 0x7f70400fcfb0 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7040112fb0 con 0x7f70400fcfb0 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7030004290 con 0x7f70400fcfb0 2024-03-24T11:16:35.384 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f703001c630 con 0x7f70400fcfb0 2024-03-24T11:16:35.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.382+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7040113540 con 0x7f70400fcfb0 2024-03-24T11:16:35.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.383+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f704004c710 con 0x7f70400fcfb0 2024-03-24T11:16:35.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.383+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f703000bbf0 con 0x7f70400fcfb0 2024-03-24T11:16:35.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.384+0000 7f70367fc700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f702c0692a0 0x7f702c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:35.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.384+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7030087af0 con 0x7f70400fcfb0 2024-03-24T11:16:35.385 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.384+0000 7f7045ee3700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f702c0692a0 0x7f702c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:35.386 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.384+0000 7f7045ee3700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f702c0692a0 0x7f702c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f703c007fd0 tx=0x7f703c009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:35.388 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.388+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7030057bf0 con 0x7f70400fcfb0 2024-03-24T11:16:35.604 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:35.603+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7f7040101320 con 0x7f702c0692a0 2024-03-24T11:16:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:36 smithi105 ceph-mon[36437]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:36 smithi105 ceph-mon[36437]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:36 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-24T11:16:36.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:36 smithi100 ceph-mon[28941]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:36.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:36 smithi100 ceph-mon[28941]: from='client.14464 -' entity='client.admin' cmd=[{"prefix": "fs volume create", "name": "cephfs", "placement": "4", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:16:36.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:36 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]: dispatch 2024-03-24T11:16:37.278 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.278+0000 7f70367fc700 1 -- 172.21.15.100:0/640782440 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7040101320 con 0x7f702c0692a0 2024-03-24T11:16:37.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f702c0692a0 msgr2=0x7f702c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:37.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f702c0692a0 0x7f702c06b750 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f703c007fd0 tx=0x7f703c009040 comp rx=0 tx=0).stop 2024-03-24T11:16:37.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 msgr2=0x7f7040112730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:37.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f7040112730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f7030009150 tx=0x7f7030009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:37.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 shutdown_connections 2024-03-24T11:16:37.281 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f702c0692a0 0x7f702c06b750 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:37.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70400fbd00 0x7f7040066cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:37.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 --2- 172.21.15.100:0/640782440 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f70400fcfb0 0x7f7040112730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:37.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 >> 172.21.15.100:0/640782440 conn(0x7f70400f7480 msgr2=0x7f70400f8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:37.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 shutdown_connections 2024-03-24T11:16:37.282 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:37.281+0000 7f7048147700 1 -- 172.21.15.100:0/640782440 wait complete. 2024-03-24T11:16:37.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:37 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-24T11:16:37.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:37 smithi105 ceph-mon[36437]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:16:37.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:37 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-24T11:16:37.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:37 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "osd pool create", "pool": "cephfs.cephfs.meta"}]': finished 2024-03-24T11:16:37.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:37 smithi100 ceph-mon[28941]: osdmap e42: 6 total, 6 up, 6 in 2024-03-24T11:16:37.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:37 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]: dispatch 2024-03-24T11:16:37.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:16:37.254+0000 7fa3e7450700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:16:37.977 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T11:16:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: pgmap v172: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:16:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: fsmap cephfs:0 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: Saving service mds.cephfs spec with placement count:4 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.begydc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:38.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.begydc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:38.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:38.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: Deploying daemon mds.cephfs.smithi100.begydc on smithi100 2024-03-24T11:16:38.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:38.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: pgmap v172: 33 pgs: 3 creating+peering, 1 active+clean, 29 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: Health check failed: 1 pool(s) do not have an application enabled (POOL_APP_NOT_ENABLED) 2024-03-24T11:16:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"bulk": true, "prefix": "osd pool create", "pool": "cephfs.cephfs.data"}]': finished 2024-03-24T11:16:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: osdmap e43: 6 total, 6 up, 6 in 2024-03-24T11:16:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]: dispatch 2024-03-24T11:16:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: Health check failed: 1 filesystem is online with fewer MDS than max_mds (MDS_UP_LESS_THAN_MAX) 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: osdmap e44: 6 total, 6 up, 6 in 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs.cephfs.meta", "data": "cephfs.cephfs.data"}]': finished 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: fsmap cephfs:0 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: Saving service mds.cephfs spec with placement count:4 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.begydc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.begydc", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:38.644 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: Deploying daemon mds.cephfs.smithi100.begydc on smithi100 2024-03-24T11:16:38.645 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:38.645 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:39.166 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:39.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:39 smithi100 ceph-mon[28941]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:16:39.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:39 smithi105 ceph-mon[36437]: osdmap e45: 6 total, 6 up, 6 in 2024-03-24T11:16:40.454 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:40 smithi100 ceph-mon[28941]: pgmap v176: 65 pgs: 23 active+clean, 9 creating+peering, 33 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:40.454 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:40 smithi100 ceph-mon[28941]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:16:40.454 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:40 smithi100 ceph-mon[28941]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T11:16:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:40 smithi105 ceph-mon[36437]: pgmap v176: 65 pgs: 23 active+clean, 9 creating+peering, 33 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:40 smithi105 ceph-mon[36437]: Health check cleared: POOL_APP_NOT_ENABLED (was: 1 pool(s) do not have an application enabled) 2024-03-24T11:16:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:40 smithi105 ceph-mon[36437]: osdmap e46: 6 total, 6 up, 6 in 2024-03-24T11:16:41.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 -- 172.21.15.100:0/189907777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fcf80 msgr2=0x7f7be00fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:41.210 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 --2- 172.21.15.100:0/189907777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fcf80 0x7f7be00fd3f0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc004940 tx=0x7f7bdc009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 -- 172.21.15.100:0/189907777 shutdown_connections 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 --2- 172.21.15.100:0/189907777 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fcf80 0x7f7be00fd3f0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 --2- 172.21.15.100:0/189907777 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fbcd0 0x7f7be00fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 -- 172.21.15.100:0/189907777 >> 172.21.15.100:0/189907777 conn(0x7f7be00f7430 msgr2=0x7f7be00f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 -- 172.21.15.100:0/189907777 shutdown_connections 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.192+0000 7f7be7a90700 1 -- 172.21.15.100:0/189907777 wait complete. 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be7a90700 1 Processor -- start 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be7a90700 1 -- start start 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be7a90700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 0x7f7be010c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be7a90700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fcf80 0x7f7be010ce40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be7a90700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be010d440 con 0x7f7be00fbcd0 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be7a90700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7be010dc90 con 0x7f7be00fcf80 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 0x7f7be010c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 0x7f7be010c900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53830/0 (socket says 172.21.15.100:53830) 2024-03-24T11:16:41.211 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 -- 172.21.15.100:0/583733727 learned_addr learned my addr 172.21.15.100:0/583733727 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be502b700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fcf80 0x7f7be010ce40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 -- 172.21.15.100:0/583733727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fcf80 msgr2=0x7f7be010ce40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fcf80 0x7f7be010ce40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 -- 172.21.15.100:0/583733727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7bdc005040 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be502b700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fcf80 0x7f7be010ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7be582c700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 0x7f7be010c900 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000c8c0 tx=0x7f7bd00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd000a980 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f7bd000ce60 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.193+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7bd00115e0 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.194+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7be010df70 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.194+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7be010e4b0 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.194+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7be005f5f0 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.195+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f7bd0012400 con 0x7f7be00fbcd0 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.195+0000 7f7bd67fc700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7bcc0692a0 0x7f7bcc06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:41.212 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.195+0000 7f7be502b700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7bcc0692a0 0x7f7bcc06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:41.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.195+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f7bd0087190 con 0x7f7be00fbcd0 2024-03-24T11:16:41.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.196+0000 7f7be502b700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7bcc0692a0 0x7f7bcc06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00b3b0 tx=0x7f7bdc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:41.213 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.200+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f7bd0056ff0 con 0x7f7be00fbcd0 2024-03-24T11:16:41.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.435+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7f7be010e7b0 con 0x7f7be00fbcd0 2024-03-24T11:16:41.435 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.435+0000 7f7bd67fc700 1 -- 172.21.15.100:0/583733727 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 2 v2) v1 ==== 75+0+1076 (secure 0 0 0) 0x7f7bd00126b0 con 0x7f7be00fbcd0 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:e2 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:legacy client fscid: 1 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:Filesystem 'cephfs' (1) 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:fs_name cephfs 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:epoch 2 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:flags 12 joinable allow_snaps allow_multimds_snaps 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:created 2024-03-24T11:16:37.255065+0000 2024-03-24T11:16:41.436 INFO:teuthology.orchestra.run.smithi100.stdout:modified 2024-03-24T11:16:37.255162+0000 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:tableserver 0 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:root 0 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:session_timeout 60 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:session_autoclose 300 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:max_file_size 1099511627776 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:required_client_features {} 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:last_failure 0 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:last_failure_osd_epoch 0 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:16:41.437 INFO:teuthology.orchestra.run.smithi100.stdout:max_mds 1 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:in 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:up {} 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:failed 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:damaged 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:stopped 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:data_pools [3] 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:metadata_pool 2 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:inline_data disabled 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:balancer 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout:standby_count_wanted 0 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:41.438 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:41.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7bcc0692a0 msgr2=0x7f7bcc06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:41.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7bcc0692a0 0x7f7bcc06b750 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f7bdc00b3b0 tx=0x7f7bdc009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:41.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 msgr2=0x7f7be010c900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:41.439 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 0x7f7be010c900 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f7bd000c8c0 tx=0x7f7bd00044c0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 shutdown_connections 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7bcc0692a0 0x7f7bcc06b750 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f7be00fbcd0 0x7f7be010c900 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 --2- 172.21.15.100:0/583733727 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7be00fcf80 0x7f7be010ce40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 >> 172.21.15.100:0/583733727 conn(0x7f7be00f7430 msgr2=0x7f7be00f8b60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 shutdown_connections 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:41.438+0000 7f7be7a90700 1 -- 172.21.15.100:0/583733727 wait complete. 2024-03-24T11:16:41.440 INFO:teuthology.orchestra.run.smithi100.stderr:dumped fsmap epoch 2 2024-03-24T11:16:42.076 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:42.085 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:42.085 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs max_mds 1' 2024-03-24T11:16:42.350 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: pgmap v178: 65 pgs: 53 active+clean, 9 creating+peering, 3 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/583733727' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.kkjmwl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.kkjmwl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:42.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:42 smithi100 ceph-mon[28941]: Deploying daemon mds.cephfs.smithi105.kkjmwl on smithi105 2024-03-24T11:16:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: pgmap v178: 65 pgs: 53 active+clean, 9 creating+peering, 3 unknown; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-03-24T11:16:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/583733727' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.kkjmwl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.kkjmwl", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:42.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:42 smithi105 ceph-mon[36437]: Deploying daemon mds.cephfs.smithi105.kkjmwl on smithi105 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] up:boot 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: daemon mds.cephfs.smithi100.begydc assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: Cluster is now healthy 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: fsmap cephfs:0 1 up:standby 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi100.begydc"}]: dispatch 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:creating} 2024-03-24T11:16:43.548 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:43 smithi100 ceph-mon[28941]: daemon mds.cephfs.smithi100.begydc is now active in filesystem cephfs as rank 0 2024-03-24T11:16:43.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] up:boot 2024-03-24T11:16:43.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: daemon mds.cephfs.smithi100.begydc assigned to filesystem cephfs as rank 0 (now has 1 ranks) 2024-03-24T11:16:43.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: Health check cleared: MDS_ALL_DOWN (was: 1 filesystem is offline) 2024-03-24T11:16:43.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: Health check cleared: MDS_UP_LESS_THAN_MAX (was: 1 filesystem is online with fewer MDS than max_mds) 2024-03-24T11:16:43.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: Cluster is now healthy 2024-03-24T11:16:43.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: fsmap cephfs:0 1 up:standby 2024-03-24T11:16:43.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi100.begydc"}]: dispatch 2024-03-24T11:16:43.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:creating} 2024-03-24T11:16:43.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:43 smithi105 ceph-mon[36437]: daemon mds.cephfs.smithi100.begydc is now active in filesystem cephfs as rank 0 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 -- 172.21.15.100:0/2512468941 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 msgr2=0x7fb1200ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 --2- 172.21.15.100:0/2512468941 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb1200ff9c0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7fb114004940 tx=0x7fb114009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 -- 172.21.15.100:0/2512468941 shutdown_connections 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 --2- 172.21.15.100:0/2512468941 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 0x7fb120102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 --2- 172.21.15.100:0/2512468941 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb1200ff9c0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 -- 172.21.15.100:0/2512468941 >> 172.21.15.100:0/2512468941 conn(0x7fb1200f7430 msgr2=0x7fb1200f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 -- 172.21.15.100:0/2512468941 shutdown_connections 2024-03-24T11:16:44.134 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.097+0000 7fb126ea3700 1 -- 172.21.15.100:0/2512468941 wait complete. 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb126ea3700 1 Processor -- start 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb126ea3700 1 -- start start 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb126ea3700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb12010da20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb126ea3700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 0x7fb12010df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb126ea3700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010e560 con 0x7fb1200662e0 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb126ea3700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb12010ed60 con 0x7fb1200fff00 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb124c3f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb12010da20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb124c3f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb12010da20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53852/0 (socket says 172.21.15.100:53852) 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb124c3f700 1 -- 172.21.15.100:0/719920481 learned_addr learned my addr 172.21.15.100:0/719920481 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.098+0000 7fb11ffff700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 0x7fb12010df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11ffff700 1 -- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 msgr2=0x7fb12010da20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11ffff700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb12010da20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11ffff700 1 -- 172.21.15.100:0/719920481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb114005040 con 0x7fb1200fff00 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb124c3f700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb12010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T11:16:44.135 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11ffff700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 0x7fb12010df60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009fd0 tx=0x7fb10c00b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb10c004e90 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fb10c00f040 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb10c00ca50 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb12010efe0 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.099+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12010f520 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.100+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb12005f5f0 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.101+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fb10c007cc0 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.101+0000 7fb11d7fa700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb1100692a0 0x7fb11006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.101+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fb10c086c10 con 0x7fb1200fff00 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.101+0000 7fb124c3f700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb1100692a0 0x7fb11006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.101+0000 7fb124c3f700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb1100692a0 0x7fb11006b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1140045a0 tx=0x7fb114009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:44.136 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.106+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fb10c04e190 con 0x7fb1200fff00 2024-03-24T11:16:44.378 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:44.378+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"} v 0) v1 -- 0x7fb12010f830 con 0x7fb1200fff00 2024-03-24T11:16:44.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: pgmap v179: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 520 B/s wr, 2 op/s 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] up:active 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.ccifff", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.ccifff", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:44.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: pgmap v179: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 520 B/s wr, 2 op/s 2024-03-24T11:16:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] up:active 2024-03-24T11:16:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 2024-03-24T11:16:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.ccifff", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi100.ccifff", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:45.304 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.304+0000 7fb11d7fa700 1 -- 172.21.15.100:0/719920481 <== mon.1 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]=0 v8) v1 ==== 105+0+0 (secure 0 0 0) 0x7fb10c0569e0 con 0x7fb1200fff00 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb1100692a0 msgr2=0x7fb11006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb1100692a0 0x7fb11006b750 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7fb1140045a0 tx=0x7fb114009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 msgr2=0x7fb12010df60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 0x7fb12010df60 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fb10c009fd0 tx=0x7fb10c00b040 comp rx=0 tx=0).stop 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 shutdown_connections 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fb1100692a0 0x7fb11006b750 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fb1200662e0 0x7fb12010da20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 --2- 172.21.15.100:0/719920481 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb1200fff00 0x7fb12010df60 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 >> 172.21.15.100:0/719920481 conn(0x7fb1200f7430 msgr2=0x7fb1200f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 shutdown_connections 2024-03-24T11:16:45.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:45.306+0000 7fb126ea3700 1 -- 172.21.15.100:0/719920481 wait complete. 2024-03-24T11:16:45.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: Deploying daemon mds.cephfs.smithi100.ccifff on smithi100 2024-03-24T11:16:45.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.105:6824/3313291601,v1:172.21.15.105:6825/3313291601] up:boot 2024-03-24T11:16:45.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby 2024-03-24T11:16:45.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.kkjmwl"}]: dispatch 2024-03-24T11:16:45.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby 2024-03-24T11:16:45.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/719920481' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T11:16:45.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:45 smithi100 ceph-mon[28941]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T11:16:45.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: Deploying daemon mds.cephfs.smithi100.ccifff on smithi100 2024-03-24T11:16:45.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.105:6824/3313291601,v1:172.21.15.105:6825/3313291601] up:boot 2024-03-24T11:16:45.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby 2024-03-24T11:16:45.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.kkjmwl"}]: dispatch 2024-03-24T11:16:45.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby 2024-03-24T11:16:45.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/719920481' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T11:16:45.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:45 smithi105 ceph-mon[36437]: from='client.? ' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]: dispatch 2024-03-24T11:16:46.305 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:46.332 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:46.332 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs allow_standby_replay true' 2024-03-24T11:16:46.536 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:46 smithi100 ceph-mon[28941]: pgmap v180: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 389 B/s wr, 3 op/s 2024-03-24T11:16:46.536 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:46 smithi100 ceph-mon[28941]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-24T11:16:46.536 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:46 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby 2024-03-24T11:16:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:46 smithi105 ceph-mon[36437]: pgmap v180: 65 pgs: 65 active+clean; 578 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 389 B/s wr, 3 op/s 2024-03-24T11:16:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:46 smithi105 ceph-mon[36437]: from='client.? ' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "max_mds", "val": "1"}]': finished 2024-03-24T11:16:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:46 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby 2024-03-24T11:16:46.963 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.rwkhto", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.rwkhto", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: Deploying daemon mds.cephfs.smithi105.rwkhto on smithi105 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T11:16:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.100:6828/1288525520,v1:172.21.15.100:6829/1288525520] up:boot 2024-03-24T11:16:48.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 2 up:standby 2024-03-24T11:16:48.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:47 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi100.ccifff"}]: dispatch 2024-03-24T11:16:48.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:48.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:48.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:48.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.rwkhto", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]: dispatch 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd='[{"prefix": "auth get-or-create", "entity": "mds.cephfs.smithi105.rwkhto", "caps": ["mon", "profile mds", "osd", "allow rw tag cephfs *=*", "mds", "allow"]}]': finished 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: Deploying daemon mds.cephfs.smithi105.rwkhto on smithi105 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: pgmap v181: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s wr, 4 op/s 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.100:6828/1288525520,v1:172.21.15.100:6829/1288525520] up:boot 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 2 up:standby 2024-03-24T11:16:48.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:47 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi100.ccifff"}]: dispatch 2024-03-24T11:16:48.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 -- 172.21.15.100:0/2001235384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 msgr2=0x7f47540fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:48.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 --2- 172.21.15.100:0/2001235384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f47540fd3f0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7f4748004940 tx=0x7f4748009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:48.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 -- 172.21.15.100:0/2001235384 shutdown_connections 2024-03-24T11:16:48.589 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 --2- 172.21.15.100:0/2001235384 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f47540fd3f0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 --2- 172.21.15.100:0/2001235384 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47540fbcd0 0x7f47540fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 -- 172.21.15.100:0/2001235384 >> 172.21.15.100:0/2001235384 conn(0x7f47540f7450 msgr2=0x7f47540f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 -- 172.21.15.100:0/2001235384 shutdown_connections 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.587+0000 7f4758a86700 1 -- 172.21.15.100:0/2001235384 wait complete. 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4758a86700 1 Processor -- start 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4758a86700 1 -- start start 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4758a86700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47540fbcd0 0x7f475410d910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4758a86700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f475410de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4758a86700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475410e450 con 0x7f47540fcf80 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4758a86700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475410e5c0 con 0x7f47540fbcd0 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f475410de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f475410de50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:45486/0 (socket says 172.21.15.100:45486) 2024-03-24T11:16:48.590 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 -- 172.21.15.100:0/744876022 learned_addr learned my addr 172.21.15.100:0/744876022 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f475259c700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47540fbcd0 0x7f475410d910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 -- 172.21.15.100:0/744876022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47540fbcd0 msgr2=0x7f475410d910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47540fbcd0 0x7f475410d910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 -- 172.21.15.100:0/744876022 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4748005040 con 0x7f47540fcf80 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.588+0000 7f4751d9b700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f475410de50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f474800b3b0 tx=0x7f4748009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.589+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474800e040 con 0x7f47540fcf80 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.589+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4754112fc0 con 0x7f47540fcf80 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.589+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f474800bbf0 con 0x7f47540fcf80 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.589+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474801c630 con 0x7f47540fcf80 2024-03-24T11:16:48.591 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.589+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4754113550 con 0x7f47540fcf80 2024-03-24T11:16:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.590+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f475404c710 con 0x7f47540fcf80 2024-03-24T11:16:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.590+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f474800bd60 con 0x7f47540fcf80 2024-03-24T11:16:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.590+0000 7f4742ffd700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f473c0692a0 0x7f473c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.591+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4748087e70 con 0x7f47540fcf80 2024-03-24T11:16:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.591+0000 7f475259c700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f473c0692a0 0x7f473c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:48.592 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.591+0000 7f475259c700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f473c0692a0 0x7f473c06b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f474c004fd0 tx=0x7f474c009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:48.595 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.595+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4748057c30 con 0x7f47540fcf80 2024-03-24T11:16:48.828 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:48.828+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"} v 0) v1 -- 0x7f47541058c0 con 0x7f47540fcf80 2024-03-24T11:16:49.192 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.192+0000 7f4742ffd700 1 -- 172.21.15.100:0/744876022 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]=0 v10) v1 ==== 121+0+0 (secure 0 0 0) 0x7f47480577c0 con 0x7f47540fcf80 2024-03-24T11:16:49.195 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f473c0692a0 msgr2=0x7f473c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f473c0692a0 0x7f473c06b750 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f474c004fd0 tx=0x7f474c009040 comp rx=0 tx=0).stop 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 msgr2=0x7f475410de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f475410de50 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7f474800b3b0 tx=0x7f4748009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 shutdown_connections 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f473c0692a0 0x7f473c06b750 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47540fbcd0 0x7f475410d910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 --2- 172.21.15.100:0/744876022 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f47540fcf80 0x7f475410de50 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 >> 172.21.15.100:0/744876022 conn(0x7f47540f7450 msgr2=0x7f47540f8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 shutdown_connections 2024-03-24T11:16:49.196 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:49.195+0000 7f4758a86700 1 -- 172.21.15.100:0/744876022 wait complete. 2024-03-24T11:16:49.484 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:49 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:49.484 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:49 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/744876022' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T11:16:49.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:49 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:49.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:49 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/744876022' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]: dispatch 2024-03-24T11:16:49.935 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:49.945 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:49.945 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph fs set cephfs inline_data true --yes-i-really-really-mean-it' 2024-03-24T11:16:50.458 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T11:16:50.458 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/744876022' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T11:16:50.458 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 2 up:standby 2024-03-24T11:16:50.458 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 1 up:standby 2024-03-24T11:16:50.458 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.459 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.459 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.459 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.459 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.459 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:50 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: pgmap v182: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s wr, 4 op/s 2024-03-24T11:16:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/744876022' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "allow_standby_replay", "val": "true"}]': finished 2024-03-24T11:16:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 2 up:standby 2024-03-24T11:16:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 1 up:standby 2024-03-24T11:16:50.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:50.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:50 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:50.866 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:51.202 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:51 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.105:6826/232471891,v1:172.21.15.105:6827/232471891] up:boot 2024-03-24T11:16:51.203 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:51 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:51.203 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:51 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.rwkhto"}]: dispatch 2024-03-24T11:16:51.318 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:51 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.105:6826/232471891,v1:172.21.15.105:6827/232471891] up:boot 2024-03-24T11:16:51.319 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:51 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:51.319 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:51 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.rwkhto"}]: dispatch 2024-03-24T11:16:52.256 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 258 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] up:active 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.100:6828/1288525520,v1:172.21.15.100:6829/1288525520] up:standby 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: Dropping low affinity standby-replay daemon mds.cephfs.smithi105.kkjmwl in favor of higher affinity standby. 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 1 up:standby 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:52.257 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:52 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: pgmap v183: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 258 B/s rd, 1.1 KiB/s wr, 4 op/s 2024-03-24T11:16:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] up:active 2024-03-24T11:16:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.100:6828/1288525520,v1:172.21.15.100:6829/1288525520] up:standby 2024-03-24T11:16:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: Dropping low affinity standby-replay daemon mds.cephfs.smithi105.kkjmwl in favor of higher affinity standby. 2024-03-24T11:16:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:52.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 1 up:standby 2024-03-24T11:16:52.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:52.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:52 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 -- 172.21.15.100:0/2317357151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fdfe0 msgr2=0x7f01e00fe450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 --2- 172.21.15.100:0/2317357151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fdfe0 0x7f01e00fe450 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f01d4004940 tx=0x7f01d4009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 -- 172.21.15.100:0/2317357151 shutdown_connections 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 --2- 172.21.15.100:0/2317357151 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fdfe0 0x7f01e00fe450 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 --2- 172.21.15.100:0/2317357151 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01e00fce70 0x7f01e00fd280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 -- 172.21.15.100:0/2317357151 >> 172.21.15.100:0/2317357151 conn(0x7f01e00f8550 msgr2=0x7f01e00fa960 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 -- 172.21.15.100:0/2317357151 shutdown_connections 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.072+0000 7f01e580c700 1 -- 172.21.15.100:0/2317357151 wait complete. 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01e580c700 1 Processor -- start 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01e580c700 1 -- start start 2024-03-24T11:16:53.075 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01e580c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 0x7f01e010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01e580c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01e00fdfe0 0x7f01e010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01e580c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e010e460 con 0x7f01e00fce70 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01e580c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e010e5d0 con 0x7f01e00fdfe0 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01deffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 0x7f01e010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01deffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 0x7f01e010d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:45506/0 (socket says 172.21.15.100:45506) 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01deffd700 1 -- 172.21.15.100:0/833737839 learned_addr learned my addr 172.21.15.100:0/833737839 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.073+0000 7f01de7fc700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01e00fdfe0 0x7f01e010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01deffd700 1 -- 172.21.15.100:0/833737839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01e00fdfe0 msgr2=0x7f01e010de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01deffd700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01e00fdfe0 0x7f01e010de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01deffd700 1 -- 172.21.15.100:0/833737839 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01d4005040 con 0x7f01e00fce70 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01deffd700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 0x7f01e010d920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f01d000c830 tx=0x7f01d0004470 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01d000a760 con 0x7f01e00fce70 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f01d000ad90 con 0x7f01e00fce70 2024-03-24T11:16:53.076 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e0112fd0 con 0x7f01e00fce70 2024-03-24T11:16:53.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f01d00113a0 con 0x7f01e00fce70 2024-03-24T11:16:53.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.074+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e0113560 con 0x7f01e00fce70 2024-03-24T11:16:53.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.075+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e004c710 con 0x7f01e00fce70 2024-03-24T11:16:53.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.075+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f01d0012400 con 0x7f01e00fce70 2024-03-24T11:16:53.077 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.076+0000 7f01c77fe700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f01c80692a0 0x7f01c806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:53.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.076+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f01d00871f0 con 0x7f01e00fce70 2024-03-24T11:16:53.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.076+0000 7f01de7fc700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f01c80692a0 0x7f01c806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:53.078 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.076+0000 7f01de7fc700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f01c80692a0 0x7f01c806b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f01d400b3b0 tx=0x7f01d4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:53.081 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.080+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f01d0056fb0 con 0x7f01e00fce70 2024-03-24T11:16:53.323 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:53 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.105:6824/3853005512,v1:172.21.15.105:6825/3853005512] up:boot 2024-03-24T11:16:53.323 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:53 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:53.323 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.kkjmwl"}]: dispatch 2024-03-24T11:16:53.323 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:53.323 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:53.323 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:53.334 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:53.333+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f01e01137b0 con 0x7f01e00fce70 2024-03-24T11:16:53.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:53 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.105:6824/3853005512,v1:172.21.15.105:6825/3853005512] up:boot 2024-03-24T11:16:53.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:53 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:53.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mds metadata", "who": "cephfs.smithi105.kkjmwl"}]: dispatch 2024-03-24T11:16:53.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:53.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:16:53.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:54.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.213+0000 7f01c77fe700 1 -- 172.21.15.100:0/833737839 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]=0 inline data enabled v16) v1 ==== 168+0+0 (secure 0 0 0) 0x7f01d005c070 con 0x7f01e00fce70 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f01c80692a0 msgr2=0x7f01c806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f01c80692a0 0x7f01c806b750 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f01d400b3b0 tx=0x7f01d4009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 msgr2=0x7f01e010d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 0x7f01e010d920 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f01d000c830 tx=0x7f01d0004470 comp rx=0 tx=0).stop 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 shutdown_connections 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f01c80692a0 0x7f01c806b750 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f01e00fce70 0x7f01e010d920 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 --2- 172.21.15.100:0/833737839 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f01e00fdfe0 0x7f01e010de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 >> 172.21.15.100:0/833737839 conn(0x7f01e00f8550 msgr2=0x7f01e00f9c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 shutdown_connections 2024-03-24T11:16:54.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:54.215+0000 7f01e580c700 1 -- 172.21.15.100:0/833737839 wait complete. 2024-03-24T11:16:54.217 INFO:teuthology.orchestra.run.smithi100.stderr:inline data enabled 2024-03-24T11:16:54.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:54 smithi100 ceph-mon[28941]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-24T11:16:54.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:54 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/833737839' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:16:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:54 smithi105 ceph-mon[36437]: pgmap v184: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 1.1 KiB/s wr, 5 op/s 2024-03-24T11:16:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:54 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/833737839' entity='client.admin' cmd=[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]: dispatch 2024-03-24T11:16:55.080 INFO:teuthology.run_tasks:Running task cephadm.shell... 2024-03-24T11:16:55.091 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:16:55.091 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph fs dump' 2024-03-24T11:16:55.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:55 smithi105 ceph-mon[36437]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-24T11:16:55.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:55 smithi105 ceph-mon[36437]: mds.? [v2:172.21.15.105:6826/232471891,v1:172.21.15.105:6827/232471891] up:standby 2024-03-24T11:16:55.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:55 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/833737839' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:16:55.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:55 smithi105 ceph-mon[36437]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:55.366 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:55 smithi100 ceph-mon[28941]: Health check failed: 1 filesystem with deprecated feature inline_data (FS_INLINE_DATA_DEPRECATED) 2024-03-24T11:16:55.366 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:55 smithi100 ceph-mon[28941]: mds.? [v2:172.21.15.105:6826/232471891,v1:172.21.15.105:6827/232471891] up:standby 2024-03-24T11:16:55.367 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:55 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/833737839' entity='client.admin' cmd='[{"prefix": "fs set", "fs_name": "cephfs", "var": "inline_data", "val": "true", "yes_i_really_really_mean_it": true}]': finished 2024-03-24T11:16:55.367 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:55 smithi100 ceph-mon[28941]: fsmap cephfs:1 {0=cephfs.smithi100.begydc=up:active} 1 up:standby-replay 2 up:standby 2024-03-24T11:16:55.821 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:16:56.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:56 smithi105 ceph-mon[36437]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 853 B/s wr, 4 op/s 2024-03-24T11:16:56.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:56 smithi100 ceph-mon[28941]: pgmap v185: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.5 KiB/s rd, 853 B/s wr, 4 op/s 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 -- 172.21.15.100:0/1590947191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 msgr2=0x7fe05c0ffa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1590947191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0ffa30 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7fe04c004940 tx=0x7fe04c009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 -- 172.21.15.100:0/1590947191 shutdown_connections 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1590947191 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 0x7fe05c102350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1590947191 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0ffa30 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 -- 172.21.15.100:0/1590947191 >> 172.21.15.100:0/1590947191 conn(0x7fe05c0f7480 msgr2=0x7fe05c0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 -- 172.21.15.100:0/1590947191 shutdown_connections 2024-03-24T11:16:57.503 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.478+0000 7fe0610f2700 1 -- 172.21.15.100:0/1590947191 wait complete. 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe0610f2700 1 Processor -- start 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe0610f2700 1 -- start start 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe0610f2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe0610f2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 0x7fe05c0fb930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe0610f2700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0fbee0 con 0x7fe05c066b10 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe0610f2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe05c0fc050 con 0x7fe05c0fff70 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 0x7fe05c0fb930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05ad9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05a59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 0x7fe05c0fb930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.100:50464/0 (socket says 172.21.15.100:50464) 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05a59c700 1 -- 172.21.15.100:0/1157879222 learned_addr learned my addr 172.21.15.100:0/1157879222 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05ad9d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:45522/0 (socket says 172.21.15.100:45522) 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05ad9d700 1 -- 172.21.15.100:0/1157879222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 msgr2=0x7fe05c0fb930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:57.504 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05ad9d700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 0x7fe05c0fb930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.479+0000 7fe05ad9d700 1 -- 172.21.15.100:0/1157879222 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe04c005040 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.480+0000 7fe05ad9d700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0fd460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe04c000c00 tx=0x7fe04c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.480+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04c00e040 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.480+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe05c0fc2d0 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.480+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fe04c00bbf0 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.480+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe04c013660 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.480+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe05c115a90 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.481+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe05c05f5f0 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.481+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fe04c00bd60 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.482+0000 7fe0437fe700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0440692a0 0x7fe04406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.482+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fe04c088560 con 0x7fe05c066b10 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.482+0000 7fe05a59c700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0440692a0 0x7fe04406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.482+0000 7fe05a59c700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0440692a0 0x7fe04406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0500041d0 tx=0x7fe05000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:16:57.505 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.486+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fe04c054890 con 0x7fe05c066b10 2024-03-24T11:16:57.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.731+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump"} v 0) v1 -- 0x7fe05c115ce0 con 0x7fe05c066b10 2024-03-24T11:16:57.733 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.732+0000 7fe0437fe700 1 -- 172.21.15.100:0/1157879222 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump"}]=0 dumped fsmap epoch 16 v16) v1 ==== 76+0+1839 (secure 0 0 0) 0x7fe04c057eb0 con 0x7fe05c066b10 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:e16 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:enable_multiple, ever_enabled_multiple: 1,1 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:legacy client fscid: 1 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:Filesystem 'cephfs' (1) 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:fs_name cephfs 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:epoch 16 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:flags 32 joinable allow_snaps allow_multimds_snaps allow_standby_replay 2024-03-24T11:16:57.734 INFO:teuthology.orchestra.run.smithi100.stdout:created 2024-03-24T11:16:37.255065+0000 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:modified 2024-03-24T11:16:54.210537+0000 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:tableserver 0 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:root 0 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:session_timeout 60 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:session_autoclose 300 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:max_file_size 1099511627776 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:required_client_features {} 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:last_failure 0 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:last_failure_osd_epoch 0 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:compat compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,7=mds uses inline data,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:max_mds 1 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:in 0 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:up {0=14472} 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:failed 2024-03-24T11:16:57.735 INFO:teuthology.orchestra.run.smithi100.stdout:damaged 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:stopped 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:data_pools [3] 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:metadata_pool 2 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:inline_data enabled 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:balancer 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:standby_count_wanted 1 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.cephfs.smithi100.begydc{0:14472} state up:active seq 4 join_fscid=1 addr [v2:172.21.15.100:6826/1495545115,v1:172.21.15.100:6827/1495545115] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.cephfs.smithi100.ccifff{0:14482} state up:standby-replay seq 2 join_fscid=1 addr [v2:172.21.15.100:6828/1288525520,v1:172.21.15.100:6829/1288525520] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:Standby daemons: 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.cephfs.smithi105.rwkhto{-1:24261} state up:standby seq 2 join_fscid=1 addr [v2:172.21.15.105:6826/232471891,v1:172.21.15.105:6827/232471891] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:16:57.736 INFO:teuthology.orchestra.run.smithi100.stdout:[mds.cephfs.smithi105.kkjmwl{-1:24265} state up:standby seq 1 join_fscid=1 addr [v2:172.21.15.105:6824/3853005512,v1:172.21.15.105:6825/3853005512] compat {c=[1],r=[1],i=[7ff]}] 2024-03-24T11:16:57.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0440692a0 msgr2=0x7fe04406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:57.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0440692a0 0x7fe04406b750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7fe0500041d0 tx=0x7fe05000b040 comp rx=0 tx=0).stop 2024-03-24T11:16:57.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 msgr2=0x7fe05c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:16:57.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0fd460 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7fe04c000c00 tx=0x7fe04c009e30 comp rx=0 tx=0).stop 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 shutdown_connections 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fe0440692a0 0x7fe04406b750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fe05c066b10 0x7fe05c0fd460 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 --2- 172.21.15.100:0/1157879222 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe05c0fff70 0x7fe05c0fb930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 >> 172.21.15.100:0/1157879222 conn(0x7fe05c0f7480 msgr2=0x7fe05c0f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 shutdown_connections 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:16:57.734+0000 7fe0610f2700 1 -- 172.21.15.100:0/1157879222 wait complete. 2024-03-24T11:16:57.738 INFO:teuthology.orchestra.run.smithi100.stderr:dumped fsmap epoch 16 2024-03-24T11:16:58.373 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T11:16:58.373 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:58.374 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:58.374 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:58.374 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:58.374 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:58.374 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:58.374 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:16:58 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/1157879222' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T11:16:58.399 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'ceph --format=json fs dump | jq -e ".filesystems | length == 1"' 2024-03-24T11:16:58.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: pgmap v186: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 853 B/s wr, 3 op/s 2024-03-24T11:16:58.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:58.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:58.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:16:58.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:16:58.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:16:58.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:16:58.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:16:58 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/1157879222' entity='client.admin' cmd=[{"prefix": "fs dump"}]: dispatch 2024-03-24T11:16:59.005 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:00.721 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:00 smithi100 ceph-mon[28941]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:17:00.721 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:00.721 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:00 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:00.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:00 smithi105 ceph-mon[36437]: pgmap v187: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:17:00.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:00 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:00.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:00 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:00.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.817+0000 7f0498e9c700 1 -- 172.21.15.100:0/2519742246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fe060 msgr2=0x7f04940fe4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:00.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.817+0000 7f0498e9c700 1 --2- 172.21.15.100:0/2519742246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fe060 0x7f04940fe4d0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7f0488004940 tx=0x7f0488009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:00.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- 172.21.15.100:0/2519742246 shutdown_connections 2024-03-24T11:17:00.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 --2- 172.21.15.100:0/2519742246 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fe060 0x7f04940fe4d0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:00.818 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 --2- 172.21.15.100:0/2519742246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fcef0 0x7f04940fd300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:00.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- 172.21.15.100:0/2519742246 >> 172.21.15.100:0/2519742246 conn(0x7f04940f85b0 msgr2=0x7f04940fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:00.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- 172.21.15.100:0/2519742246 shutdown_connections 2024-03-24T11:17:00.819 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- 172.21.15.100:0/2519742246 wait complete. 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 Processor -- start 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- start start 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 0x7f049410d9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fe060 0x7f049410df00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049410e500 con 0x7f04940fcef0 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.818+0000 7f0498e9c700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f049410ed50 con 0x7f04940fe060 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 0x7f049410d9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 0x7f049410d9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60254/0 (socket says 172.21.15.100:60254) 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 -- 172.21.15.100:0/651088788 learned_addr learned my addr 172.21.15.100:0/651088788 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0491d9b700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fe060 0x7f049410df00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 -- 172.21.15.100:0/651088788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fe060 msgr2=0x7f049410df00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:00.821 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fe060 0x7f049410df00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 -- 172.21.15.100:0/651088788 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0488005040 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0491d9b700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fe060 0x7f049410df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f049259c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 0x7f049410d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f048400eb40 tx=0x7f048400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0484007cc0 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f049410efd0 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f048400f040 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0484009620 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.819+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f049410f510 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.820+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f049405f5f0 con 0x7f04940fcef0 2024-03-24T11:17:00.822 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.821+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0484009780 con 0x7f04940fcef0 2024-03-24T11:17:00.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.821+0000 7f0482ffd700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f047c0692a0 0x7f047c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:00.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.821+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0484086e00 con 0x7f04940fcef0 2024-03-24T11:17:00.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.821+0000 7f0491d9b700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f047c0692a0 0x7f047c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:00.823 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.821+0000 7f0491d9b700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f047c0692a0 0x7f047c06b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0488009b30 tx=0x7f0488009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:00.826 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:00.826+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f048404e190 con 0x7f04940fcef0 2024-03-24T11:17:01.060 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.060+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f049404bed0 con 0x7f04940fcef0 2024-03-24T11:17:01.061 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.061+0000 7f0482ffd700 1 -- 172.21.15.100:0/651088788 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4703 (secure 0 0 0) 0x7f0484056bf0 con 0x7f04940fcef0 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.062+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f047c0692a0 msgr2=0x7f047c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.062+0000 7f0498e9c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f047c0692a0 0x7f047c06b750 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f0488009b30 tx=0x7f0488009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.062+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 msgr2=0x7f049410d9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.062+0000 7f0498e9c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 0x7f049410d9c0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7f048400eb40 tx=0x7f048400b040 comp rx=0 tx=0).stop 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 shutdown_connections 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f047c0692a0 0x7f047c06b750 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:01.063 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f04940fcef0 0x7f049410d9c0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 --2- 172.21.15.100:0/651088788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f04940fe060 0x7f049410df00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 >> 172.21.15.100:0/651088788 conn(0x7f04940f85b0 msgr2=0x7f04940f9bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 shutdown_connections 2024-03-24T11:17:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:01.063+0000 7f0498e9c700 1 -- 172.21.15.100:0/651088788 wait complete. 2024-03-24T11:17:01.064 INFO:teuthology.orchestra.run.smithi100.stderr:dumped fsmap epoch 16 2024-03-24T11:17:01.073 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-03-24T11:17:01.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:01 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/651088788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T11:17:01.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:01 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/651088788' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T11:17:01.934 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- bash -c 'while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done' 2024-03-24T11:17:02.604 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:02.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:02 smithi100 ceph-mon[28941]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:17:02.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:02 smithi105 ceph-mon[36437]: pgmap v188: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.2 KiB/s rd, 0 B/s wr, 3 op/s 2024-03-24T11:17:04.708 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:04 smithi100 ceph-mon[28941]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-03-24T11:17:04.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:04 smithi105 ceph-mon[36437]: pgmap v189: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.3 KiB/s rd, 3 op/s 2024-03-24T11:17:05.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 -- 172.21.15.100:0/847126612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e700fd2c0 msgr2=0x7f2e700fd6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:05.264 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 --2- 172.21.15.100:0/847126612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e700fd2c0 0x7f2e700fd6d0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009ce0 tx=0x7f2e68004d10 comp rx=0 tx=0).stop 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 -- 172.21.15.100:0/847126612 shutdown_connections 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 --2- 172.21.15.100:0/847126612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e700fdc10 0x7f2e700fc030 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 --2- 172.21.15.100:0/847126612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e700fd2c0 0x7f2e700fd6d0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 -- 172.21.15.100:0/847126612 >> 172.21.15.100:0/847126612 conn(0x7f2e700f6bd0 msgr2=0x7f2e700f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 -- 172.21.15.100:0/847126612 shutdown_connections 2024-03-24T11:17:05.265 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 -- 172.21.15.100:0/847126612 wait complete. 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 Processor -- start 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.262+0000 7f2e748df700 1 -- start start 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e748df700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e700fd2c0 0x7f2e700fb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e748df700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 0x7f2e70112c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e748df700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701131b0 con 0x7f2e70110890 2024-03-24T11:17:05.266 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e748df700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e701132f0 con 0x7f2e700fd2c0 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 0x7f2e70112c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 0x7f2e70112c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60282/0 (socket says 172.21.15.100:60282) 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 -- 172.21.15.100:0/191203910 learned_addr learned my addr 172.21.15.100:0/191203910 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6e59c700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e700fd2c0 0x7f2e700fb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 -- 172.21.15.100:0/191203910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e700fd2c0 msgr2=0x7f2e700fb940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e700fd2c0 0x7f2e700fb940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 -- 172.21.15.100:0/191203910 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e68009940 con 0x7f2e70110890 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e6dd9b700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 0x7f2e70112c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2e64002a50 tx=0x7f2e640046b0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e64007cc0 con 0x7f2e70110890 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e70113580 con 0x7f2e70110890 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2e6400f040 con 0x7f2e70110890 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2e6400d9d0 con 0x7f2e70110890 2024-03-24T11:17:05.267 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.263+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e70113b40 con 0x7f2e70110890 2024-03-24T11:17:05.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.264+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e7004c710 con 0x7f2e70110890 2024-03-24T11:17:05.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.265+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2e6400dc40 con 0x7f2e70110890 2024-03-24T11:17:05.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.265+0000 7f2e5effd700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e580692a0 0x7f2e5806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:05.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.265+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2e64086f30 con 0x7f2e70110890 2024-03-24T11:17:05.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.265+0000 7f2e6e59c700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e580692a0 0x7f2e5806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:05.268 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.265+0000 7f2e6e59c700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e580692a0 0x7f2e5806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009470 tx=0x7f2e68004d10 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:05.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.269+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2e64056c70 con 0x7f2e70110890 2024-03-24T11:17:05.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.541+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "mds versions", "format": "json"} v 0) v1 -- 0x7f2e701047e0 con 0x7f2e70110890 2024-03-24T11:17:05.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.541+0000 7f2e5effd700 1 -- 172.21.15.100:0/191203910 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "mds versions", "format": "json"}]=0 v16) v1 ==== 78+0+99 (secure 0 0 0) 0x7f2e64056800 con 0x7f2e70110890 2024-03-24T11:17:05.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e580692a0 msgr2=0x7f2e5806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:05.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e580692a0 0x7f2e5806b750 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f2e68009470 tx=0x7f2e68004d10 comp rx=0 tx=0).stop 2024-03-24T11:17:05.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 msgr2=0x7f2e70112c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:05.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 0x7f2e70112c70 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f2e64002a50 tx=0x7f2e640046b0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.544 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 shutdown_connections 2024-03-24T11:17:05.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2e580692a0 0x7f2e5806b750 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e700fd2c0 0x7f2e700fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 --2- 172.21.15.100:0/191203910 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2e70110890 0x7f2e70112c70 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:05.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.543+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 >> 172.21.15.100:0/191203910 conn(0x7f2e700f6bd0 msgr2=0x7f2e700f82e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:05.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.544+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 shutdown_connections 2024-03-24T11:17:05.545 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:05.544+0000 7f2e748df700 1 -- 172.21.15.100:0/191203910 wait complete. 2024-03-24T11:17:05.554 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-03-24T11:17:06.144 INFO:teuthology.run_tasks:Running task fs.pre_upgrade_save... 2024-03-24T11:17:06.180 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T11:17:06.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3387525516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 msgr2=0x7f6bd80fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3387525516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd80fee10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f6bd40048d0 tx=0x7f6bd4009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.307 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3387525516 shutdown_connections 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3387525516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6bd80ff350 0x7f6bd8101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3387525516 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd80fee10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3387525516 >> 172.21.15.100:0/3387525516 conn(0x7f6bd80f6840 msgr2=0x7f6bd80f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3387525516 shutdown_connections 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3387525516 wait complete. 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 Processor -- start 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.305+0000 7f6bdf62e640 1 -- start start 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdf62e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd810cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdf62e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6bd80ff350 0x7f6bd810d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdf62e640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd810d8d0 con 0x7f6bd8064fd0 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdf62e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bd8112290 con 0x7f6bd80ff350 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd810cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd810cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60300/0 (socket says 172.21.15.100:60300) 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 -- 172.21.15.100:0/3491700230 learned_addr learned my addr 172.21.15.100:0/3491700230 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:06.308 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdcbc9640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6bd80ff350 0x7f6bd810d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 -- 172.21.15.100:0/3491700230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6bd80ff350 msgr2=0x7f6bd810d280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6bd80ff350 0x7f6bd810d280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 -- 172.21.15.100:0/3491700230 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bd4005040 con 0x7f6bd8064fd0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.306+0000 7f6bdd3ca640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd810cd40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4003080 tx=0x7f6bd4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.307+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd40093b0 con 0x7f6bd8064fd0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.307+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bd81123d0 con 0x7f6bd8064fd0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.307+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6bd4009510 con 0x7f6bd8064fd0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.307+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6bd4043620 con 0x7f6bd8064fd0 2024-03-24T11:17:06.309 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.307+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bd81128a0 con 0x7f6bd8064fd0 2024-03-24T11:17:06.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.308+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bd804bcc0 con 0x7f6bd8064fd0 2024-03-24T11:17:06.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.308+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6bd4043890 con 0x7f6bd8064fd0 2024-03-24T11:17:06.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.309+0000 7f6bcdffb640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6bc4069210 0x7f6bc406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.309+0000 7f6bdcbc9640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6bc4069210 0x7f6bc406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.309+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6bd40aa260 con 0x7f6bd8064fd0 2024-03-24T11:17:06.310 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.309+0000 7f6bdcbc9640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6bc4069210 0x7f6bc406b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8007fd0 tx=0x7f6bc8009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:06.314 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.313+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6bd4076530 con 0x7f6bd8064fd0 2024-03-24T11:17:06.493 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.493+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6bd8104d00 con 0x7f6bd8064fd0 2024-03-24T11:17:06.494 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.493+0000 7f6bcdffb640 1 -- 172.21.15.100:0/3491700230 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v46) v1 ==== 74+0+14320 (secure 0 0 0) 0x7f6bd4079b50 con 0x7f6bd8064fd0 2024-03-24T11:17:06.494 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:17:06.494 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":46,"fsid":"e2f9ef6c-e9ce-11ee-95cd-87774f69a715","created":"2024-03-24T11:09:42.710984+0000","modified":"2024-03-24T11:16:39.263189+0000","last_up_change":"2024-03-24T11:15:17.863076+0000","last_in_change":"2024-03-24T11:14:58.878641+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":3,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-03-24T11:14:05.424502+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}},{"pool":2,"pool_name":"cephfs.cephfs.meta","create_time":"2024-03-24T11:16:35.839563+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"45","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":3,"pool_name":"cephfs.cephfs.data","create_time":"2024-03-24T11:16:36.482049+0000","flags":131073,"flags_names":"hashpspool,bulk","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":32,"pg_placement_num":32,"pg_placement_num_target":32,"pg_num_target":32,"pg_num_pending":32,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"46","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"cc6e0f08-af11-46b2-923f-17e549042b98","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6802","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6803","nonce":3447712544}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6804","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6805","nonce":3447712544}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6808","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6809","nonce":3447712544}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6806","nonce":3447712544},{"type":"v1","addr":"172.21.15.100:6807","nonce":3447712544}]},"public_addr":"172.21.15.100:6803/3447712544","cluster_addr":"172.21.15.100:6805/3447712544","heartbeat_back_addr":"172.21.15.100:6809/3447712544","heartbeat_front_addr":"172.21.15.100:6807/3447712544","state":["exists","up"]},{"osd":1,"uuid":"3cfba2e6-a0a2-44af-8a5c-ea2d5910f5cb","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6810","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6811","nonce":1490052239}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6812","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6813","nonce":1490052239}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6816","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6817","nonce":1490052239}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6814","nonce":1490052239},{"type":"v1","addr":"172.21.15.100:6815","nonce":1490052239}]},"public_addr":"172.21.15.100:6811/1490052239","cluster_addr":"172.21.15.100:6813/1490052239","heartbeat_back_addr":"172.21.15.100:6817/1490052239","heartbeat_front_addr":"172.21.15.100:6815/1490052239","state":["exists","up"]},{"osd":2,"uuid":"44db3f0c-2877-4026-93f3-febdb7f46d67","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6818","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6819","nonce":2671131686}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6820","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6821","nonce":2671131686}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6824","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6825","nonce":2671131686}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6822","nonce":2671131686},{"type":"v1","addr":"172.21.15.100:6823","nonce":2671131686}]},"public_addr":"172.21.15.100:6819/2671131686","cluster_addr":"172.21.15.100:6821/2671131686","heartbeat_back_addr":"172.21.15.100:6825/2671131686","heartbeat_front_addr":"172.21.15.100:6823/2671131686","state":["exists","up"]},{"osd":3,"uuid":"1f6ededc-bddf-4308-a416-80ae19ad3fa4","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6801","nonce":3221138489}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6803","nonce":3221138489}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6807","nonce":3221138489}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":3221138489},{"type":"v1","addr":"172.21.15.105:6805","nonce":3221138489}]},"public_addr":"172.21.15.105:6801/3221138489","cluster_addr":"172.21.15.105:6803/3221138489","heartbeat_back_addr":"172.21.15.105:6807/3221138489","heartbeat_front_addr":"172.21.15.105:6805/3221138489","state":["exists","up"]},{"osd":4,"uuid":"c59e7571-57a0-406f-bdbb-614deeaaa56e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6809","nonce":3756920843}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6811","nonce":3756920843}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6815","nonce":3756920843}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3756920843},{"type":"v1","addr":"172.21.15.105:6813","nonce":3756920843}]},"public_addr":"172.21.15.105:6809/3756920843","cluster_addr":"172.21.15.105:6811/3756920843","heartbeat_back_addr":"172.21.15.105:6815/3756920843","heartbeat_front_addr":"172.21.15.105:6813/3756920843","state":["exists","up"]},{"osd":5,"uuid":"15b5ed12-6a0d-4bfe-9ffd-5224e21353ae","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":43,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6817","nonce":256673904}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6819","nonce":256673904}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6823","nonce":256673904}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":256673904},{"type":"v1","addr":"172.21.15.105:6821","nonce":256673904}]},"public_addr":"172.21.15.105:6817/256673904","cluster_addr":"172.21.15.105:6819/256673904","heartbeat_back_addr":"172.21.15.105:6823/256673904","heartbeat_front_addr":"172.21.15.105:6821/256673904","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:03.013039+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:13:32.779377+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:01.889286+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:25.258963+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:14:50.202419+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-03-24T11:15:16.423165+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.100:0/2476598760":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/3850505170":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6801/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/374004208":"2024-03-25T11:10:47.648919+0000","172.21.15.100:6801/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/3084685173":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/1681068182":"2024-03-25T11:10:16.917673+0000","172.21.15.100:0/3561410096":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2555446076":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6801/1648043756":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1493018813":"2024-03-25T11:11:53.102021+0000","172.21.15.100:6800/1790161450":"2024-03-25T11:10:47.648919+0000","172.21.15.100:0/2493584253":"2024-03-25T11:11:53.102021+0000","172.21.15.100:0/4249885134":"2024-03-25T11:10:16.917673+0000","172.21.15.100:6800/1648043756":"2024-03-25T11:10:16.917673+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-03-24T11:17:06.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6bc4069210 msgr2=0x7f6bc406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6bc4069210 0x7f6bc406b6c0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f6bc8007fd0 tx=0x7f6bc8009040 comp rx=0 tx=0).stop 2024-03-24T11:17:06.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 msgr2=0x7f6bd810cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd810cd40 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f6bd4003080 tx=0x7f6bd4009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 shutdown_connections 2024-03-24T11:17:06.496 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6bc4069210 0x7f6bc406b6c0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6bd80ff350 0x7f6bd810d280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 --2- 172.21.15.100:0/3491700230 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6bd8064fd0 0x7f6bd810cd40 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.495+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 >> 172.21.15.100:0/3491700230 conn(0x7f6bd80f6840 msgr2=0x7f6bd80f6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:06.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.496+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 shutdown_connections 2024-03-24T11:17:06.497 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.496+0000 7f6bdf62e640 1 -- 172.21.15.100:0/3491700230 wait complete. 2024-03-24T11:17:06.507 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-03-24T11:14:05.424502+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}, {'pool': 2, 'pool_name': 'cephfs.cephfs.meta', 'create_time': '2024-03-24T11:16:35.839563+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '45', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 3, 'pool_name': 'cephfs.cephfs.data', 'create_time': '2024-03-24T11:16:36.482049+0000', 'flags': 131073, 'flags_names': 'hashpspool,bulk', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 32, 'pg_placement_num': 32, 'pg_placement_num_target': 32, 'pg_num_target': 32, 'pg_num_pending': 32, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '46', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2024-03-24T11:17:06.507 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get .mgr pg_num 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 -- 172.21.15.100:0/436029818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 msgr2=0x7f4dd00fb4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 --2- 172.21.15.100:0/436029818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd00fb4d0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f4dc00048d0 tx=0x7f4dc0009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 -- 172.21.15.100:0/436029818 shutdown_connections 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 --2- 172.21.15.100:0/436029818 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4dd00fc390 0x7f4dd00fc800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 --2- 172.21.15.100:0/436029818 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd00fb4d0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 -- 172.21.15.100:0/436029818 >> 172.21.15.100:0/436029818 conn(0x7f4dd00f6880 msgr2=0x7f4dd00f8cb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:06.611 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 -- 172.21.15.100:0/436029818 shutdown_connections 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 -- 172.21.15.100:0/436029818 wait complete. 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.608+0000 7f4dd611d640 1 Processor -- start 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 -- start start 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd010ccb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4dd00fc390 0x7f4dd010d1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd010d870 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4dd0112230 con 0x7f4dd00fc390 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd010ccb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd010ccb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60314/0 (socket says 172.21.15.100:60314) 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 -- 172.21.15.100:0/3302952674 learned_addr learned my addr 172.21.15.100:0/3302952674 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dceffd640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4dd00fc390 0x7f4dd010d1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.612 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 -- 172.21.15.100:0/3302952674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4dd00fc390 msgr2=0x7f4dd010d1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4dd00fc390 0x7f4dd010d1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 -- 172.21.15.100:0/3302952674 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4dc0005040 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dcf7fe640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd010ccb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0007d50 tx=0x7f4dc0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dc0009590 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4dc00096f0 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4dd0112370 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4dc003b6d0 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.609+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4dd0112860 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.610+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4dd004bcc0 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.613 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.611+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4dc002f070 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.611+0000 7f4db7fff640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4db8069210 0x7f4db806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.611+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4dc00a98a0 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.614 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.611+0000 7f4dceffd640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4db8069210 0x7f4db806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.615 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.612+0000 7f4dceffd640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4db8069210 0x7f4db806b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4dd010e1a0 tx=0x7f4dc4009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:06.616 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.615+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4dc0075b50 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.782 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.782+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f4dd01129f0 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.782+0000 7f4db7fff640 1 -- 172.21.15.100:0/3302952674 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v46) v1 ==== 93+0+10 (secure 0 0 0) 0x7f4dc0079170 con 0x7f4dd00fb0c0 2024-03-24T11:17:06.783 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 1 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4db8069210 msgr2=0x7f4db806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4db8069210 0x7f4db806b6c0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f4dd010e1a0 tx=0x7f4dc4009040 comp rx=0 tx=0).stop 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 msgr2=0x7f4dd010ccb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd010ccb0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f4dc0007d50 tx=0x7f4dc0009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 shutdown_connections 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4db8069210 0x7f4db806b6c0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4dd00fc390 0x7f4dd010d1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 --2- 172.21.15.100:0/3302952674 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4dd00fb0c0 0x7f4dd010ccb0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 >> 172.21.15.100:0/3302952674 conn(0x7f4dd00f6880 msgr2=0x7f4dd00f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 shutdown_connections 2024-03-24T11:17:06.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.784+0000 7f4dd611d640 1 -- 172.21.15.100:0/3302952674 wait complete. 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/191203910' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:06.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:06 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:06.795 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.meta pg_num 2024-03-24T11:17:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: pgmap v190: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:17:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/191203910' entity='client.admin' cmd=[{"prefix": "mds versions", "format": "json"}]: dispatch 2024-03-24T11:17:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:17:06.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:17:06.803 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:06 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:06.896 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 -- 172.21.15.100:0/1023153895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 msgr2=0x7f6c8c0fee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 --2- 172.21.15.100:0/1023153895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c0fee20 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f6c880048d0 tx=0x7f6c88009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 -- 172.21.15.100:0/1023153895 shutdown_connections 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 --2- 172.21.15.100:0/1023153895 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c8c0ff360 0x7f6c8c101740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 --2- 172.21.15.100:0/1023153895 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c0fee20 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 -- 172.21.15.100:0/1023153895 >> 172.21.15.100:0/1023153895 conn(0x7f6c8c0f6870 msgr2=0x7f6c8c0f8c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 -- 172.21.15.100:0/1023153895 shutdown_connections 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.894+0000 7f6c9366e640 1 -- 172.21.15.100:0/1023153895 wait complete. 2024-03-24T11:17:06.897 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9366e640 1 Processor -- start 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9366e640 1 -- start start 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9366e640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c10cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9366e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c8c0ff360 0x7f6c8c10d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9366e640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c10d8e0 con 0x7f6c8c064fb0 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9366e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c8c1122a0 con 0x7f6c8c0ff360 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9140a640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c10cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9140a640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c10cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60322/0 (socket says 172.21.15.100:60322) 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9140a640 1 -- 172.21.15.100:0/327487747 learned_addr learned my addr 172.21.15.100:0/327487747 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c90c09640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c8c0ff360 0x7f6c8c10d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9140a640 1 -- 172.21.15.100:0/327487747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c8c0ff360 msgr2=0x7f6c8c10d290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9140a640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c8c0ff360 0x7f6c8c10d290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:06.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.895+0000 7f6c9140a640 1 -- 172.21.15.100:0/327487747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c88005040 con 0x7f6c8c064fb0 2024-03-24T11:17:06.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.896+0000 7f6c9140a640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c10cd50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6c8802bda0 tx=0x7f6c88009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:06.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.896+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c880094c0 con 0x7f6c8c064fb0 2024-03-24T11:17:06.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.896+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f6c88009620 con 0x7f6c8c064fb0 2024-03-24T11:17:06.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.896+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c8c1123e0 con 0x7f6c8c064fb0 2024-03-24T11:17:06.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.896+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6c8803b640 con 0x7f6c8c064fb0 2024-03-24T11:17:06.899 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.896+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c8c1128b0 con 0x7f6c8c064fb0 2024-03-24T11:17:06.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.897+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c8c04bcc0 con 0x7f6c8c064fb0 2024-03-24T11:17:06.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.897+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f6c8802f070 con 0x7f6c8c064fb0 2024-03-24T11:17:06.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.898+0000 7f6c81ffb640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6c78069210 0x7f6c7806b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:06.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.898+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f6c880a97a0 con 0x7f6c8c064fb0 2024-03-24T11:17:06.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.898+0000 7f6c90c09640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6c78069210 0x7f6c7806b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:06.900 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.898+0000 7f6c90c09640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6c78069210 0x7f6c7806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c007fd0 tx=0x7f6c7c009e90 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:06.902 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:06.902+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f6c88075a50 con 0x7f6c8c064fb0 2024-03-24T11:17:07.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.067+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"} v 0) v1 -- 0x7f6c8c112a40 con 0x7f6c8c064fb0 2024-03-24T11:17:07.068 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.068+0000 7f6c81ffb640 1 -- 172.21.15.100:0/327487747 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f6c88079070 con 0x7f6c8c064fb0 2024-03-24T11:17:07.068 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 32 2024-03-24T11:17:07.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6c78069210 msgr2=0x7f6c7806b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6c78069210 0x7f6c7806b6c0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f6c7c007fd0 tx=0x7f6c7c009e90 comp rx=0 tx=0).stop 2024-03-24T11:17:07.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 msgr2=0x7f6c8c10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.070 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c10cd50 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f6c8802bda0 tx=0x7f6c88009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 shutdown_connections 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f6c78069210 0x7f6c7806b6c0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6c8c0ff360 0x7f6c8c10d290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 --2- 172.21.15.100:0/327487747 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f6c8c064fb0 0x7f6c8c10cd50 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 >> 172.21.15.100:0/327487747 conn(0x7f6c8c0f6870 msgr2=0x7f6c8c0f6c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 shutdown_connections 2024-03-24T11:17:07.071 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.070+0000 7f6c9366e640 1 -- 172.21.15.100:0/327487747 wait complete. 2024-03-24T11:17:07.082 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs.cephfs.data pg_num 2024-03-24T11:17:07.184 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 -- 172.21.15.100:0/713075729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fc1c0 msgr2=0x7f79840fc5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 --2- 172.21.15.100:0/713075729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fc1c0 0x7f79840fc5d0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f79800048d0 tx=0x7f7980009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 -- 172.21.15.100:0/713075729 shutdown_connections 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 --2- 172.21.15.100:0/713075729 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fd490 0x7f79840fd900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 --2- 172.21.15.100:0/713075729 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fc1c0 0x7f79840fc5d0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 -- 172.21.15.100:0/713075729 >> 172.21.15.100:0/713075729 conn(0x7f79840f7960 msgr2=0x7f79840f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 -- 172.21.15.100:0/713075729 shutdown_connections 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.182+0000 7f798c75d640 1 -- 172.21.15.100:0/713075729 wait complete. 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798c75d640 1 Processor -- start 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798c75d640 1 -- start start 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798c75d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fc1c0 0x7f798410ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798c75d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 0x7f798410d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:07.185 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798c75d640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f798410d890 con 0x7f79840fd490 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798c75d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7984112250 con 0x7f79840fc1c0 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 0x7f798410d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 0x7f798410d210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60344/0 (socket says 172.21.15.100:60344) 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 -- 172.21.15.100:0/868549720 learned_addr learned my addr 172.21.15.100:0/868549720 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798a4f9640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fc1c0 0x7f798410ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 -- 172.21.15.100:0/868549720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fc1c0 msgr2=0x7f798410ccd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fc1c0 0x7f798410ccd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 -- 172.21.15.100:0/868549720 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7980005040 con 0x7f79840fd490 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f798a4f9640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fc1c0 0x7f798410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f7989cf8640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 0x7f798410d210 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7974002a80 tx=0x7f7974004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.183+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7974007c30 con 0x7f79840fd490 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.184+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f797400f040 con 0x7f79840fd490 2024-03-24T11:17:07.186 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.184+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79841123f0 con 0x7f79840fd490 2024-03-24T11:17:07.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.184+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f797400db80 con 0x7f79840fd490 2024-03-24T11:17:07.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.184+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79841128e0 con 0x7f79840fd490 2024-03-24T11:17:07.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.185+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f798404bcc0 con 0x7f79840fd490 2024-03-24T11:17:07.187 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.185+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f797400dd20 con 0x7f79840fd490 2024-03-24T11:17:07.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.185+0000 7f797affd640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7970069210 0x7f797006b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:07.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.185+0000 7f798a4f9640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7970069210 0x7f797006b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:07.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.185+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f79740869e0 con 0x7f79840fd490 2024-03-24T11:17:07.188 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.186+0000 7f798a4f9640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7970069210 0x7f797006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7980000c00 tx=0x7f7980009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:07.190 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.189+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f797404d190 con 0x7f79840fd490 2024-03-24T11:17:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.356+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"} v 0) v1 -- 0x7f7984112b80 con 0x7f79840fd490 2024-03-24T11:17:07.357 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.357+0000 7f797affd640 1 -- 172.21.15.100:0/868549720 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]=0 v46) v1 ==== 107+0+11 (secure 0 0 0) 0x7f7974052dd0 con 0x7f79840fd490 2024-03-24T11:17:07.357 INFO:teuthology.orchestra.run.smithi100.stdout:pg_num: 32 2024-03-24T11:17:07.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7970069210 msgr2=0x7f797006b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7970069210 0x7f797006b6c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7980000c00 tx=0x7f7980009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.359 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 msgr2=0x7f798410d210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 0x7f798410d210 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f7974002a80 tx=0x7f7974004600 comp rx=0 tx=0).stop 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 shutdown_connections 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f79840fd490 0x7f798410d210 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f7970069210 0x7f797006b6c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 --2- 172.21.15.100:0/868549720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79840fc1c0 0x7f798410ccd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 >> 172.21.15.100:0/868549720 conn(0x7f79840f7960 msgr2=0x7f79840f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 shutdown_connections 2024-03-24T11:17:07.360 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.359+0000 7f798c75d640 1 -- 172.21.15.100:0/868549720 wait complete. 2024-03-24T11:17:07.370 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2024-03-24T11:17:07.469 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.466+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/2189646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a38064fd0 msgr2=0x7f2a380fee10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.466+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/2189646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a38064fd0 0x7f2a380fee10 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f2a280048d0 tx=0x7f2a28009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.466+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/2189646 shutdown_connections 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.466+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/2189646 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a380ff350 0x7f2a38101730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.466+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/2189646 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a38064fd0 0x7f2a380fee10 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.466+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/2189646 >> 172.21.15.100:0/2189646 conn(0x7f2a380f6840 msgr2=0x7f2a380f8c90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/2189646 shutdown_connections 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/2189646 wait complete. 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 Processor -- start 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 -- start start 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a38064fd0 0x7f2a3810cd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 0x7f2a3810d2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:07.470 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3810d8f0 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3ff5d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a381122b0 con 0x7f2a38064fd0 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3d4f8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 0x7f2a3810d2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3d4f8640 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 0x7f2a3810d2a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:60364/0 (socket says 172.21.15.100:60364) 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3d4f8640 1 -- 172.21.15.100:0/748299008 learned_addr learned my addr 172.21.15.100:0/748299008 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3dcf9640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a38064fd0 0x7f2a3810cd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3d4f8640 1 -- 172.21.15.100:0/748299008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a38064fd0 msgr2=0x7f2a3810cd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3d4f8640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a38064fd0 0x7f2a3810cd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.467+0000 7f2a3d4f8640 1 -- 172.21.15.100:0/748299008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a28005040 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.468+0000 7f2a3d4f8640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 0x7f2a3810d2a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400eb70 tx=0x7f2a3400b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.468+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a34007c30 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.468+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f2a3400f040 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.468+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a381123f0 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.468+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2a340095b0 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.468+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a381128c0 con 0x7f2a380ff350 2024-03-24T11:17:07.471 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.469+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3804bcc0 con 0x7f2a380ff350 2024-03-24T11:17:07.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.469+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f2a34009730 con 0x7f2a380ff350 2024-03-24T11:17:07.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.469+0000 7f2a2e7fc640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2a24069210 0x7f2a2406b6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:07.472 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.469+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f2a34086f00 con 0x7f2a380ff350 2024-03-24T11:17:07.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.469+0000 7f2a3dcf9640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2a24069210 0x7f2a2406b6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:07.473 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.470+0000 7f2a3dcf9640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2a24069210 0x7f2a2406b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2a28000c00 tx=0x7f2a28009da0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:07.474 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.474+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f2a34053270 con 0x7f2a380ff350 2024-03-24T11:17:07.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3491700230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:17:07.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3302952674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:17:07.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/327487747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-24T11:17:07.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:07 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/868549720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-24T11:17:07.675 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.675+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2a38112ae0 con 0x7f2a380ff350 2024-03-24T11:17:07.676 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.676+0000 7f2a2e7fc640 1 -- 172.21.15.100:0/748299008 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 16 v16) v1 ==== 94+0+4703 (secure 0 0 0) 0x7f2a34056890 con 0x7f2a380ff350 2024-03-24T11:17:07.676 INFO:teuthology.orchestra.run.smithi100.stdout: 2024-03-24T11:17:07.676 INFO:teuthology.orchestra.run.smithi100.stdout:{"epoch":16,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":true,"ever_enabled_multiple":true},"standbys":[{"gid":24261,"name":"cephfs.smithi105.rwkhto","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.105:6827/232471891","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6826","nonce":232471891},{"type":"v1","addr":"172.21.15.105:6827","nonce":232471891}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":16},{"gid":24265,"name":"cephfs.smithi105.kkjmwl","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.105:6825/3853005512","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3853005512},{"type":"v1","addr":"172.21.15.105:6825","nonce":3853005512}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"epoch":15}],"filesystems":[{"mdsmap":{"epoch":16,"flags":50,"flags_state":{"joinable":true,"allow_snaps":true,"allow_multimds_snaps":true,"allow_standby_replay":true},"ever_allowed_features":32,"explicitly_allowed_features":32,"created":"2024-03-24T11:16:37.255065+0000","modified":"2024-03-24T11:16:54.210537+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"required_client_features":{},"max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":14472},"failed":[],"damaged":[],"stopped":[],"info":{"gid_14472":{"gid":14472,"name":"cephfs.smithi100.begydc","rank":0,"incarnation":4,"state":"up:active","state_seq":4,"addr":"172.21.15.100:6827/1495545115","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6826","nonce":1495545115},{"type":"v1","addr":"172.21.15.100:6827","nonce":1495545115}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}},"gid_14482":{"gid":14482,"name":"cephfs.smithi100.ccifff","rank":0,"incarnation":0,"state":"up:standby-replay","state_seq":2,"addr":"172.21.15.100:6829/1288525520","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.100:6828","nonce":1288525520},{"type":"v1","addr":"172.21.15.100:6829","nonce":1288525520}]},"join_fscid":1,"export_targets":[],"features":4540138320759226367,"flags":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_7":"mds uses inline data","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}}}},"data_pools":[3],"metadata_pool":2,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":1},"id":1}]} 2024-03-24T11:17:07.677 INFO:teuthology.orchestra.run.smithi100.stderr:dumped fsmap epoch 16 2024-03-24T11:17:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2a24069210 msgr2=0x7f2a2406b6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2a24069210 0x7f2a2406b6c0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f2a28000c00 tx=0x7f2a28009da0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.678 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 msgr2=0x7f2a3810d2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 0x7f2a3810d2a0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f2a3400eb70 tx=0x7f2a3400b040 comp rx=0 tx=0).stop 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 shutdown_connections 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f2a380ff350 0x7f2a3810d2a0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f2a24069210 0x7f2a2406b6c0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 --2- 172.21.15.100:0/748299008 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2a38064fd0 0x7f2a3810cd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 >> 172.21.15.100:0/748299008 conn(0x7f2a380f6840 msgr2=0x7f2a380f6c80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 shutdown_connections 2024-03-24T11:17:07.679 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:07.678+0000 7f2a3ff5d640 1 -- 172.21.15.100:0/748299008 wait complete. 2024-03-24T11:17:07.689 DEBUG:tasks.fs:fs fscid=1,name=cephfs state = {'epoch': 16, 'max_mds': 1, 'flags': 50} 2024-03-24T11:17:07.689 INFO:teuthology.run_tasks:Running task kclient... 2024-03-24T11:17:07.771 INFO:tasks.kclient:Mounting kernel clients... 2024-03-24T11:17:07.772 INFO:tasks.kclient:config is {'client.0': {}, 'client.1': {}} 2024-03-24T11:17:07.772 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:07.772 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-03-24T11:17:07.787 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:07.788 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link delete ceph-brx 2024-03-24T11:17:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:07 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3491700230' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-03-24T11:17:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:07 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3302952674' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-03-24T11:17:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:07 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/327487747' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.meta", "var": "pg_num"}]: dispatch 2024-03-24T11:17:07.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:07 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/868549720' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": "cephfs.cephfs.data", "var": "pg_num"}]: dispatch 2024-03-24T11:17:07.856 INFO:teuthology.orchestra.run.smithi100.stderr:Cannot find device "ceph-brx" 2024-03-24T11:17:07.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:17:07.858 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:07.858 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-03-24T11:17:07.877 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:07.877 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link delete ceph-brx 2024-03-24T11:17:07.953 INFO:teuthology.orchestra.run.smithi105.stderr:Cannot find device "ceph-brx" 2024-03-24T11:17:07.954 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:17:07.955 INFO:tasks.kclient:client.0 config is {'syntax': 'v1'} 2024-03-24T11:17:07.955 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T11:17:07.955 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:07.955 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T11:17:08.209 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:08.573 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:08 smithi105 ceph-mon[36437]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:17:08.573 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:08 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/748299008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T11:17:08.574 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:08.574 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:08 smithi100 ceph-mon[28941]: pgmap v191: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.1 KiB/s rd, 1 op/s 2024-03-24T11:17:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:08 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/748299008' entity='client.admin' cmd=[{"prefix": "fs dump", "format": "json"}]: dispatch 2024-03-24T11:17:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:17:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:09.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 -- 172.21.15.100:0/1211621283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 msgr2=0x7fc2c41031d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:09.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 --2- 172.21.15.100:0/1211621283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c41031d0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc009ce0 tx=0x7fc2bc004d10 comp rx=0 tx=0).stop 2024-03-24T11:17:09.872 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 -- 172.21.15.100:0/1211621283 shutdown_connections 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 --2- 172.21.15.100:0/1211621283 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c41037a0 0x7fc2c40fce10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 --2- 172.21.15.100:0/1211621283 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c41031d0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 -- 172.21.15.100:0/1211621283 >> 172.21.15.100:0/1211621283 conn(0x7fc2c40f89b0 msgr2=0x7fc2c40fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 -- 172.21.15.100:0/1211621283 shutdown_connections 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.869+0000 7fc2c9951700 1 -- 172.21.15.100:0/1211621283 wait complete. 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c9951700 1 Processor -- start 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c9951700 1 -- start start 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c9951700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c4113b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c9951700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c41037a0 0x7fc2c4114050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c9951700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c4114730 con 0x7fc2c4102e00 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c9951700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2c410dc80 con 0x7fc2c41037a0 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c4113b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c2ffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c4113b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36898/0 (socket says 172.21.15.100:36898) 2024-03-24T11:17:09.873 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.870+0000 7fc2c2ffd700 1 -- 172.21.15.100:0/2976289075 learned_addr learned my addr 172.21.15.100:0/2976289075 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:09.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c2ffd700 1 -- 172.21.15.100:0/2976289075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c41037a0 msgr2=0x7fc2c4114050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:09.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c27fc700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c41037a0 0x7fc2c4114050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:09.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c2ffd700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c41037a0 0x7fc2c4114050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:09.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c2ffd700 1 -- 172.21.15.100:0/2976289075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2bc009940 con 0x7fc2c4102e00 2024-03-24T11:17:09.874 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c2ffd700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c4113b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005010 tx=0x7fc2bc004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:09.875 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2bc00e040 con 0x7fc2c4102e00 2024-03-24T11:17:09.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc2bc008e20 con 0x7fc2c4102e00 2024-03-24T11:17:09.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc2c410dea0 con 0x7fc2c4102e00 2024-03-24T11:17:09.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc2bc01c660 con 0x7fc2c4102e00 2024-03-24T11:17:09.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.871+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc2c410e460 con 0x7fc2c4102e00 2024-03-24T11:17:09.876 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.872+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc2c40fe550 con 0x7fc2c4102e00 2024-03-24T11:17:09.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.872+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc2bc012a70 con 0x7fc2c4102e00 2024-03-24T11:17:09.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.873+0000 7fc2ab7fe700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc2ac0692a0 0x7fc2ac06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:09.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.873+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc2bc088de0 con 0x7fc2c4102e00 2024-03-24T11:17:09.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.873+0000 7fc2c27fc700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc2ac0692a0 0x7fc2ac06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:09.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.873+0000 7fc2c27fc700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc2ac0692a0 0x7fc2ac06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8006fd0 tx=0x7fc2b8008040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:09.877 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:09.877+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc2bc058b20 con 0x7fc2c4102e00 2024-03-24T11:17:10.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.103+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc2c410f190 con 0x7fc2c4102e00 2024-03-24T11:17:10.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.104+0000 7fc2ab7fe700 1 -- 172.21.15.100:0/2976289075 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fc2bc0586b0 con 0x7fc2c4102e00 2024-03-24T11:17:10.104 INFO:teuthology.orchestra.run.smithi100.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.105+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc2ac0692a0 msgr2=0x7fc2ac06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.105+0000 7fc2c9951700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc2ac0692a0 0x7fc2ac06b750 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc2b8006fd0 tx=0x7fc2b8008040 comp rx=0 tx=0).stop 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.105+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 msgr2=0x7fc2c4113b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.105+0000 7fc2c9951700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c4113b10 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7fc2bc005010 tx=0x7fc2bc004d10 comp rx=0 tx=0).stop 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 shutdown_connections 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc2ac0692a0 0x7fc2ac06b750 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc2c4102e00 0x7fc2c4113b10 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 --2- 172.21.15.100:0/2976289075 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2c41037a0 0x7fc2c4114050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:10.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 >> 172.21.15.100:0/2976289075 conn(0x7fc2c40f89b0 msgr2=0x7fc2c40f9900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:10.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 shutdown_connections 2024-03-24T11:17:10.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:10.106+0000 7fc2c9951700 1 -- 172.21.15.100:0/2976289075 wait complete. 2024-03-24T11:17:10.751 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:10 smithi100 ceph-mon[28941]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:10.751 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:10 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2976289075' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi100.front.sepia.ceph.com 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:self.client.name = client.0 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T11:17:10.777 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T11:17:10.778 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.0' 2024-03-24T11:17:10.778 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:10.778 DEBUG:teuthology.orchestra.run.smithi100:> ip addr 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 ::1/128 scope host 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:28 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:10.795 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f0 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:d8:a1:29 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: altname enp7s0f1 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:32 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: link/ether 0c:c4:7a:8f:d7:33 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: inet 172.21.15.100/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft 42285sec preferred_lft 42285sec 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: inet6 fe80::ec4:7aff:fe8f:d733/64 scope link 2024-03-24T11:17:10.796 INFO:teuthology.orchestra.run.smithi100.stdout: valid_lft forever preferred_lft forever 2024-03-24T11:17:10.797 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T11:17:10.797 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:10.797 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-03-24T11:17:10.798 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-03-24T11:17:10.798 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add name ceph-brx type bridge 2024-03-24T11:17:10.798 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip addr flush dev ceph-brx 2024-03-24T11:17:10.798 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set ceph-brx up 2024-03-24T11:17:10.798 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T11:17:10.798 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-03-24T11:17:10.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:10 smithi105 ceph-mon[36437]: pgmap v192: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:10.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:10 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2976289075' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T11:17:10.964 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:10.964 DEBUG:teuthology.orchestra.run.smithi100:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T11:17:10.991 INFO:teuthology.orchestra.run.smithi100.stdout:1 2024-03-24T11:17:10.992 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:10.992 DEBUG:teuthology.orchestra.run.smithi100:> route 2024-03-24T11:17:11.066 INFO:teuthology.orchestra.run.smithi100.stdout:Kernel IP routing table 2024-03-24T11:17:11.066 INFO:teuthology.orchestra.run.smithi100.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T11:17:11.066 INFO:teuthology.orchestra.run.smithi100.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T11:17:11.066 INFO:teuthology.orchestra.run.smithi100.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T11:17:11.066 INFO:teuthology.orchestra.run.smithi100.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T11:17:11.067 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.067 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-03-24T11:17:11.067 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-03-24T11:17:11.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T11:17:11.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T11:17:11.067 DEBUG:teuthology.orchestra.run.smithi100:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T11:17:11.067 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-03-24T11:17:11.339 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.339 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list 2024-03-24T11:17:11.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.354 DEBUG:teuthology.orchestra.run.smithi100:> ip netns list-id 2024-03-24T11:17:11.409 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.409 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-03-24T11:17:11.409 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-03-24T11:17:11.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.0 2024-03-24T11:17:11.409 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.0 0 2024-03-24T11:17:11.409 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-03-24T11:17:11.493 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.0' with 192.168.0.1/16 2024-03-24T11:17:11.493 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.0 type veth peer name brx.0 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set veth0 up 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip link set lo up 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.0 ip route add default via 192.168.255.254 2024-03-24T11:17:11.493 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-03-24T11:17:11.630 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.630 DEBUG:teuthology.orchestra.run.smithi100:> (cd / && exec bash -c ' 2024-03-24T11:17:11.630 DEBUG:teuthology.orchestra.run.smithi100:> set -e 2024-03-24T11:17:11.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set brx.0 up 2024-03-24T11:17:11.630 DEBUG:teuthology.orchestra.run.smithi100:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T11:17:11.630 DEBUG:teuthology.orchestra.run.smithi100:> ') 2024-03-24T11:17:11.672 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T11:17:11.672 DEBUG:teuthology.orchestra.run.smithi100:> mkdir -p -v /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:11.727 INFO:teuthology.orchestra.run.smithi100.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.0' 2024-03-24T11:17:11.727 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T11:17:11.727 DEBUG:teuthology.orchestra.run.smithi100:> chmod 0000 /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:11.781 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T11:17:11.782 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.782 DEBUG:teuthology.orchestra.run.smithi100:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.0 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T11:17:11.867 INFO:teuthology.orchestra.run.smithi100.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T11:17:11.868 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T11:17:11.868 INFO:teuthology.orchestra.run.smithi100.stdout:invalid new device string format 2024-03-24T11:17:11.979 INFO:teuthology.orchestra.run.smithi100.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T11:17:11.979 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T11:17:11.979 INFO:teuthology.orchestra.run.smithi100.stdout:invalid new device string format 2024-03-24T11:17:11.980 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: resolved to: "172.21.15.100:3300,172.21.15.105:3300" 2024-03-24T11:17:11.980 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T11:17:11.984 INFO:teuthology.orchestra.run.smithi100.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=0,mds_namespace=cephfs 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs". 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:invalid new device string format 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: resolved to: "172.21.15.100:3300,172.21.15.105:3300" 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: resolved to: "172.21.15.100:6789,172.21.15.105:6789" 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.100:6789,172.21.15.105:6789:/ 2024-03-24T11:17:11.985 INFO:teuthology.orchestra.run.smithi100.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=0,mds_namespace=cephfs,key=0,fsid=e2f9ef6c-e9ce-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T11:17:11.986 INFO:teuthology.orchestra.run.smithi100.stdout:mount: /home/ubuntu/cephtest/mnt.0 does not contain SELinux labels. 2024-03-24T11:17:11.986 INFO:teuthology.orchestra.run.smithi100.stdout: You just mounted a file system that supports labels which does not 2024-03-24T11:17:11.986 INFO:teuthology.orchestra.run.smithi100.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T11:17:11.986 INFO:teuthology.orchestra.run.smithi100.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T11:17:11.986 INFO:teuthology.orchestra.run.smithi100.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T11:17:11.987 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T11:17:11.988 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:11.988 DEBUG:teuthology.orchestra.run.smithi100:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:12.015 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:12.015 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T11:17:12.075 INFO:teuthology.orchestra.run.smithi100.stdout:client24297 2024-03-24T11:17:12.075 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:17:12.075 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:17:12.131 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:12.132 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T11:17:12.192 INFO:teuthology.orchestra.run.smithi100.stdout:client24297 2024-03-24T11:17:12.192 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:12.192 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T11:17:12.252 INFO:teuthology.orchestra.run.smithi100.stdout:client24297 2024-03-24T11:17:12.252 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:12.252 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest/mnt.0 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715.client24297/status') 2024-03-24T11:17:12.322 INFO:teuthology.orchestra.run.smithi100.stdout:instance: client.24297 (3)172.21.15.100:0/1813231102 2024-03-24T11:17:12.322 INFO:teuthology.orchestra.run.smithi100.stdout:blocklisted: false 2024-03-24T11:17:12.323 INFO:teuthology.orchestra.run.smithi100.stderr:0+1 records in 2024-03-24T11:17:12.323 INFO:teuthology.orchestra.run.smithi100.stderr:0+1 records out 2024-03-24T11:17:12.323 INFO:teuthology.orchestra.run.smithi100.stderr:72 bytes copied, 3.8539e-05 s, 1.9 MB/s 2024-03-24T11:17:12.323 INFO:tasks.kclient:client.1 config is {'syntax': 'v1'} 2024-03-24T11:17:12.324 INFO:tasks.cephfs.mount:cephfs_mntpt = None 2024-03-24T11:17:12.324 INFO:tasks.cephfs.mount:hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:12.324 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -- ceph fs ls 2024-03-24T11:17:12.619 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:12.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:12 smithi100 ceph-mon[28941]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:12.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:12 smithi105 ceph-mon[36437]: pgmap v193: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.211+0000 7fc8b556f700 1 -- 172.21.15.100:0/475261593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b01037a0 msgr2=0x7fc8b00fce10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.211+0000 7fc8b556f700 1 --2- 172.21.15.100:0/475261593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b01037a0 0x7fc8b00fce10 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fc8a4004940 tx=0x7fc8a4009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- 172.21.15.100:0/475261593 shutdown_connections 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 --2- 172.21.15.100:0/475261593 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b01037a0 0x7fc8b00fce10 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 --2- 172.21.15.100:0/475261593 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8b0102e00 0x7fc8b01031d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- 172.21.15.100:0/475261593 >> 172.21.15.100:0/475261593 conn(0x7fc8b00f89b0 msgr2=0x7fc8b00fadc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:14.214 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- 172.21.15.100:0/475261593 shutdown_connections 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- 172.21.15.100:0/475261593 wait complete. 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 Processor -- start 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- start start 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 0x7fc8b0113980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8b01037a0 0x7fc8b0113ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b0114510 con 0x7fc8b0102e00 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.212+0000 7fc8b556f700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8b010db10 con 0x7fc8b01037a0 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 0x7fc8b0113980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 0x7fc8b0113980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:36914/0 (socket says 172.21.15.100:36914) 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 -- 172.21.15.100:0/2899284238 learned_addr learned my addr 172.21.15.100:0/2899284238 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8ae7fc700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8b01037a0 0x7fc8b0113ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 -- 172.21.15.100:0/2899284238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8b01037a0 msgr2=0x7fc8b0113ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8b01037a0 0x7fc8b0113ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 -- 172.21.15.100:0/2899284238 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8a4005040 con 0x7fc8b0102e00 2024-03-24T11:17:14.215 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8aeffd700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 0x7fc8b0113980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000c8c0 tx=0x7fc8a00044c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:14.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a000a980 con 0x7fc8b0102e00 2024-03-24T11:17:14.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8b010ddf0 con 0x7fc8b0102e00 2024-03-24T11:17:14.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fc8a000ce60 con 0x7fc8b0102e00 2024-03-24T11:17:14.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8a00115e0 con 0x7fc8b0102e00 2024-03-24T11:17:14.216 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.213+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8b010e3b0 con 0x7fc8b0102e00 2024-03-24T11:17:14.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.214+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8b01052d0 con 0x7fc8b0102e00 2024-03-24T11:17:14.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.215+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fc8a000aae0 con 0x7fc8b0102e00 2024-03-24T11:17:14.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.215+0000 7fc8977fe700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc8980692a0 0x7fc89806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:14.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.215+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fc8a00872a0 con 0x7fc8b0102e00 2024-03-24T11:17:14.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.215+0000 7fc8ae7fc700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc8980692a0 0x7fc89806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:14.217 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.216+0000 7fc8ae7fc700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc8980692a0 0x7fc89806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400b3b0 tx=0x7fc8a4009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:14.220 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.220+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fc8a0057060 con 0x7fc8b0102e00 2024-03-24T11:17:14.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.454+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "fs ls"} v 0) v1 -- 0x7fc8b010efd0 con 0x7fc8b0102e00 2024-03-24T11:17:14.455 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.455+0000 7fc8977fe700 1 -- 172.21.15.100:0/2899284238 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "fs ls"}]=0 v16) v1 ==== 53+0+83 (secure 0 0 0) 0x7fc8a0056bf0 con 0x7fc8b0102e00 2024-03-24T11:17:14.455 INFO:teuthology.orchestra.run.smithi100.stdout:name: cephfs, metadata pool: cephfs.cephfs.meta, data pools: [cephfs.cephfs.data ] 2024-03-24T11:17:14.457 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.456+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc8980692a0 msgr2=0x7fc89806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.456+0000 7fc8b556f700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc8980692a0 0x7fc89806b750 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fc8a400b3b0 tx=0x7fc8a4009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.456+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 msgr2=0x7fc8b0113980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.456+0000 7fc8b556f700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 0x7fc8b0113980 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fc8a000c8c0 tx=0x7fc8a00044c0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 shutdown_connections 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fc8980692a0 0x7fc89806b750 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fc8b0102e00 0x7fc8b0113980 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 --2- 172.21.15.100:0/2899284238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8b01037a0 0x7fc8b0113ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 >> 172.21.15.100:0/2899284238 conn(0x7fc8b00f89b0 msgr2=0x7fc8b01021d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 shutdown_connections 2024-03-24T11:17:14.458 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:14.457+0000 7fc8b556f700 1 -- 172.21.15.100:0/2899284238 wait complete. 2024-03-24T11:17:14.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:14 smithi105 ceph-mon[36437]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-24T11:17:14.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:14 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/2899284238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T11:17:14.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:14 smithi100 ceph-mon[28941]: pgmap v194: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 170 B/s wr, 2 op/s 2024-03-24T11:17:14.818 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:14 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/2899284238' entity='client.admin' cmd=[{"prefix": "fs ls"}]: dispatch 2024-03-24T11:17:15.125 INFO:tasks.cephfs.mount:Mounting default Ceph FS; just confirmed its presence on cluster 2024-03-24T11:17:15.125 INFO:tasks.cephfs.mount:Mounting Ceph FS. Following are details of mount; remember "None" represents Python type None - 2024-03-24T11:17:15.125 INFO:tasks.cephfs.mount:self.client_remote.hostname = smithi105.front.sepia.ceph.com 2024-03-24T11:17:15.126 INFO:tasks.cephfs.mount:self.client.name = client.1 2024-03-24T11:17:15.126 INFO:tasks.cephfs.mount:self.hostfs_mntpt = /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:15.126 INFO:tasks.cephfs.mount:self.cephfs_name = None 2024-03-24T11:17:15.126 INFO:tasks.cephfs.mount:self.cephfs_mntpt = None 2024-03-24T11:17:15.126 INFO:tasks.cephfs.mount:self.client_keyring_path = None 2024-03-24T11:17:15.126 INFO:tasks.cephfs.mount:Setting the 'None' netns for '/home/ubuntu/cephtest/mnt.1' 2024-03-24T11:17:15.127 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.127 DEBUG:teuthology.orchestra.run.smithi105:> ip addr 2024-03-24T11:17:15.146 INFO:teuthology.orchestra.run.smithi105.stdout:1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: inet 127.0.0.1/8 scope host lo 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: inet6 ::1/128 scope host 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout:2: eno1: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:d8:a4:b8 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: altname enp7s0f0 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout:3: eno2: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:d8:a4:b9 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout: altname enp7s0f1 2024-03-24T11:17:15.147 INFO:teuthology.orchestra.run.smithi105.stdout:4: enp3s0f0: mtu 1500 qdisc mq state DOWN group default qlen 1000 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:8f:d4:28 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout:5: enp3s0f1: mtu 1500 qdisc mq state UP group default qlen 1000 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout: link/ether 0c:c4:7a:8f:d4:29 brd ff:ff:ff:ff:ff:ff 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout: inet 172.21.15.105/20 brd 172.21.15.255 scope global dynamic noprefixroute enp3s0f1 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft 42264sec preferred_lft 42264sec 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout: inet6 fe80::ec4:7aff:fe8f:d429/64 scope link 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run.smithi105.stdout: valid_lft forever preferred_lft forever 2024-03-24T11:17:15.148 INFO:tasks.cephfs.mount:Setuping the 'ceph-brx' with 192.168.255.254/16 2024-03-24T11:17:15.148 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.148 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec bash -c ' 2024-03-24T11:17:15.149 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-03-24T11:17:15.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link add name ceph-brx type bridge 2024-03-24T11:17:15.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip addr flush dev ceph-brx 2024-03-24T11:17:15.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set ceph-brx up 2024-03-24T11:17:15.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip addr add 192.168.255.254/16 brd 192.168.255.255 dev ceph-brx 2024-03-24T11:17:15.149 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-03-24T11:17:15.328 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.328 DEBUG:teuthology.orchestra.run.smithi105:> echo 1 | sudo tee /proc/sys/net/ipv4/ip_forward 2024-03-24T11:17:15.356 INFO:teuthology.orchestra.run.smithi105.stdout:1 2024-03-24T11:17:15.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.358 DEBUG:teuthology.orchestra.run.smithi105:> route 2024-03-24T11:17:15.428 INFO:teuthology.orchestra.run.smithi105.stdout:Kernel IP routing table 2024-03-24T11:17:15.428 INFO:teuthology.orchestra.run.smithi105.stdout:Destination Gateway Genmask Flags Metric Ref Use Iface 2024-03-24T11:17:15.428 INFO:teuthology.orchestra.run.smithi105.stdout:default _gateway 0.0.0.0 UG 100 0 0 enp3s0f1 2024-03-24T11:17:15.429 INFO:teuthology.orchestra.run.smithi105.stdout:172.21.0.0 0.0.0.0 255.255.240.0 U 100 0 0 enp3s0f1 2024-03-24T11:17:15.429 INFO:teuthology.orchestra.run.smithi105.stdout:192.168.0.0 0.0.0.0 255.255.0.0 U 0 0 0 ceph-brx 2024-03-24T11:17:15.429 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.429 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec bash -c ' 2024-03-24T11:17:15.429 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-03-24T11:17:15.429 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -A FORWARD -o enp3s0f1 -i ceph-brx -j ACCEPT 2024-03-24T11:17:15.429 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -A FORWARD -i enp3s0f1 -o ceph-brx -j ACCEPT 2024-03-24T11:17:15.429 DEBUG:teuthology.orchestra.run.smithi105:> sudo iptables -t nat -A POSTROUTING -s 192.168.255.254/16 -o enp3s0f1 -j MASQUERADE 2024-03-24T11:17:15.429 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-03-24T11:17:15.689 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.690 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list 2024-03-24T11:17:15.704 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.704 DEBUG:teuthology.orchestra.run.smithi105:> ip netns list-id 2024-03-24T11:17:15.759 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.759 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec bash -c ' 2024-03-24T11:17:15.759 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-03-24T11:17:15.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns add ceph-ns--home-ubuntu-cephtest-mnt.1 2024-03-24T11:17:15.759 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns set ceph-ns--home-ubuntu-cephtest-mnt.1 0 2024-03-24T11:17:15.759 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-03-24T11:17:15.841 INFO:tasks.cephfs.mount:Setuping the netns 'ceph-ns--home-ubuntu-cephtest-mnt.1' with 192.168.0.1/16 2024-03-24T11:17:15.841 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec bash -c ' 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link add veth0 netns ceph-ns--home-ubuntu-cephtest-mnt.1 type veth peer name brx.0 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip addr add 192.168.0.1/16 brd 192.168.255.255 dev veth0 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set veth0 up 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip link set lo up 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip netns exec ceph-ns--home-ubuntu-cephtest-mnt.1 ip route add default via 192.168.255.254 2024-03-24T11:17:15.842 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-03-24T11:17:15.981 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:15.981 DEBUG:teuthology.orchestra.run.smithi105:> (cd / && exec bash -c ' 2024-03-24T11:17:15.981 DEBUG:teuthology.orchestra.run.smithi105:> set -e 2024-03-24T11:17:15.981 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set brx.0 up 2024-03-24T11:17:15.981 DEBUG:teuthology.orchestra.run.smithi105:> sudo ip link set dev brx.0 master ceph-brx 2024-03-24T11:17:15.982 DEBUG:teuthology.orchestra.run.smithi105:> ') 2024-03-24T11:17:16.023 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T11:17:16.023 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -v /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:16.078 INFO:teuthology.orchestra.run.smithi105.stdout:mkdir: created directory '/home/ubuntu/cephtest/mnt.1' 2024-03-24T11:17:16.078 INFO:teuthology.orchestra.run:Running command with timeout 60 2024-03-24T11:17:16.078 DEBUG:teuthology.orchestra.run.smithi105:> chmod 0000 /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:16.133 INFO:tasks.cephfs.kernel_mount:mounting using device: :/ 2024-03-24T11:17:16.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:16.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo nsenter --net=/var/run/netns/ceph-ns--home-ubuntu-cephtest-mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage /bin/mount -t ceph :/ /home/ubuntu/cephtest/mnt.1 -v -o norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T11:17:16.238 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T11:17:16.238 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T11:17:16.238 INFO:teuthology.orchestra.run.smithi105.stdout:invalid new device string format 2024-03-24T11:17:16.352 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T11:17:16.352 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T11:17:16.352 INFO:teuthology.orchestra.run.smithi105.stdout:invalid new device string format 2024-03-24T11:17:16.353 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.100:3300,172.21.15.105:3300" 2024-03-24T11:17:16.353 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T11:17:16.357 INFO:teuthology.orchestra.run.smithi105.stdout:parsing options: rw,norequire_active_mds,conf=/etc/ceph/ceph.conf,norbytes,name=1,mds_namespace=cephfs 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs". 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:invalid new device string format 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.100:3300,172.21.15.105:3300" 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: switching to using v1 address with old syntax 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: resolved to: "172.21.15.100:6789,172.21.15.105:6789" 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: trying mount with old device syntax: 172.21.15.100:6789,172.21.15.105:6789:/ 2024-03-24T11:17:16.358 INFO:teuthology.orchestra.run.smithi105.stdout:mount.ceph: options "norequire_active_mds,norbytes,name=1,mds_namespace=cephfs,key=1,fsid=e2f9ef6c-e9ce-11ee-95cd-87774f69a715" will pass to kernel 2024-03-24T11:17:16.359 INFO:teuthology.orchestra.run.smithi105.stdout:mount: /home/ubuntu/cephtest/mnt.1 does not contain SELinux labels. 2024-03-24T11:17:16.359 INFO:teuthology.orchestra.run.smithi105.stdout: You just mounted a file system that supports labels which does not 2024-03-24T11:17:16.359 INFO:teuthology.orchestra.run.smithi105.stdout: contain labels, onto an SELinux box. It is likely that confined 2024-03-24T11:17:16.359 INFO:teuthology.orchestra.run.smithi105.stdout: applications will generate AVC messages and not be allowed access to 2024-03-24T11:17:16.359 INFO:teuthology.orchestra.run.smithi105.stdout: this file system. For more details see restorecon(8) and mount(8). 2024-03-24T11:17:16.360 INFO:tasks.cephfs.kernel_mount:mount command passed 2024-03-24T11:17:16.360 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:16.361 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 1777 /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:16.387 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:16.387 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T11:17:16.443 INFO:teuthology.orchestra.run.smithi105.stdout:client14540 2024-03-24T11:17:16.444 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:17:16.444 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:17:16.499 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:16.499 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T11:17:16.556 INFO:teuthology.orchestra.run.smithi105.stdout:client14540 2024-03-24T11:17:16.556 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:16.556 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'getfattr --only-values -n ceph.client_id .') 2024-03-24T11:17:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:client14540 2024-03-24T11:17:16.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:17:16.613 DEBUG:teuthology.orchestra.run.smithi105:> (cd /home/ubuntu/cephtest/mnt.1 && exec bash -c 'sudo dd if=/sys/kernel/debug/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715.client14540/status') 2024-03-24T11:17:16.680 INFO:teuthology.orchestra.run.smithi105.stdout:instance: client.14540 (3)172.21.15.105:0/3994020011 2024-03-24T11:17:16.680 INFO:teuthology.orchestra.run.smithi105.stdout:blocklisted: false 2024-03-24T11:17:16.680 INFO:teuthology.orchestra.run.smithi105.stderr:0+1 records in 2024-03-24T11:17:16.680 INFO:teuthology.orchestra.run.smithi105.stderr:0+1 records out 2024-03-24T11:17:16.680 INFO:teuthology.orchestra.run.smithi105.stderr:72 bytes copied, 2.3021e-05 s, 3.1 MB/s 2024-03-24T11:17:16.681 INFO:teuthology.run_tasks:Running task print... 2024-03-24T11:17:16.692 INFO:teuthology.task.print:**** done client 2024-03-24T11:17:16.692 INFO:teuthology.run_tasks:Running task parallel... 2024-03-24T11:17:16.705 INFO:teuthology.task.parallel:starting parallel... 2024-03-24T11:17:16.705 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T11:17:16.706 INFO:teuthology.task.sequential:In sequential, running task cephadm.shell... 2024-03-24T11:17:16.706 INFO:tasks.cephadm:Running commands on role host.a host ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:17:16.706 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force' 2024-03-24T11:17:16.707 INFO:teuthology.task.parallel:In parallel, running task sequential... 2024-03-24T11:17:16.707 INFO:teuthology.task.sequential:In sequential, running task workunit... 2024-03-24T11:17:16.713 INFO:tasks.workunit:Pulling workunits from ref f73f1dfd96849300c4322bd7259a3ab37f94c200 2024-03-24T11:17:16.713 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-03-24T11:17:16.713 INFO:tasks.workunit:timeout=3h 2024-03-24T11:17:16.713 INFO:tasks.workunit:cleanup=True 2024-03-24T11:17:16.714 DEBUG:teuthology.orchestra.run.smithi100:> stat -- /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:16.733 INFO:teuthology.orchestra.run.smithi100.stdout: File: /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout: Size: 0 Blocks: 0 IO Block: 65536 directory 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout:Device: 94h/148d Inode: 1 Links: 2 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout:Modify: 2024-03-24 11:16:42.281278676 +0000 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout:Change: 2024-03-24 11:17:12.013569171 +0000 2024-03-24T11:17:16.734 INFO:teuthology.orchestra.run.smithi100.stdout: Birth: 2024-03-24 11:16:42.281278676 +0000 2024-03-24T11:17:16.734 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.0 2024-03-24T11:17:16.735 DEBUG:teuthology.orchestra.run.smithi100:> cd -- /home/ubuntu/cephtest/mnt.0 && sudo install -d -m 0755 --owner=ubuntu -- client.0 2024-03-24T11:17:16.743 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:16 smithi100 ceph-mon[28941]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:17:16.777 DEBUG:teuthology.orchestra.run.smithi105:> stat -- /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout: File: /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 1 Blocks: 0 IO Block: 65536 directory 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 73h/115d Inode: 1 Links: 3 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (1777/drwxrwxrwt) Uid: ( 0/ root) Gid: ( 0/ root) 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:unlabeled_t:s0 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 1970-01-01 00:00:00.000000000 +0000 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-03-24 11:17:16.771453480 +0000 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-03-24 11:17:16.771453480 +0000 2024-03-24T11:17:16.793 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-03-24 11:16:42.281278676 +0000 2024-03-24T11:17:16.794 INFO:tasks.workunit:Did not need to create dir /home/ubuntu/cephtest/mnt.1 2024-03-24T11:17:16.794 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/mnt.1 && sudo install -d -m 0755 --owner=ubuntu -- client.1 2024-03-24T11:17:16.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:16 smithi105 ceph-mon[36437]: pgmap v195: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:17:16.832 DEBUG:teuthology.orchestra.run.smithi100:> rm -rf /home/ubuntu/cephtest/clone.client.0 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.0 && cd /home/ubuntu/cephtest/clone.client.0 && git checkout f73f1dfd96849300c4322bd7259a3ab37f94c200 2024-03-24T11:17:16.833 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/clone.client.1 && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.client.1 && cd /home/ubuntu/cephtest/clone.client.1 && git checkout f73f1dfd96849300c4322bd7259a3ab37f94c200 2024-03-24T11:17:16.848 INFO:tasks.workunit.client.0.smithi100.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.0'... 2024-03-24T11:17:16.888 INFO:tasks.workunit.client.1.smithi105.stderr:Cloning into '/home/ubuntu/cephtest/clone.client.1'... 2024-03-24T11:17:16.987 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:18.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.646+0000 7faf65ca4700 1 -- 172.21.15.100:0/2453954622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 msgr2=0x7faf600fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:18.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.646+0000 7faf65ca4700 1 --2- 172.21.15.100:0/2453954622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf600fd3f0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7faf54004940 tx=0x7faf54009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:18.649 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 -- 172.21.15.100:0/2453954622 shutdown_connections 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 --2- 172.21.15.100:0/2453954622 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf600fd3f0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 --2- 172.21.15.100:0/2453954622 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf600fbcd0 0x7faf600fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 -- 172.21.15.100:0/2453954622 >> 172.21.15.100:0/2453954622 conn(0x7faf600f7430 msgr2=0x7faf600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 -- 172.21.15.100:0/2453954622 shutdown_connections 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 -- 172.21.15.100:0/2453954622 wait complete. 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 Processor -- start 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 -- start start 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf600fbcd0 0x7faf60066d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.647+0000 7faf65ca4700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf60112710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf65ca4700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60112c50 con 0x7faf600fcf80 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf65ca4700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf60112d90 con 0x7faf600fbcd0 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf60112710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:18.650 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf60112710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53832/0 (socket says 172.21.15.100:53832) 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 -- 172.21.15.100:0/3262164509 learned_addr learned my addr 172.21.15.100:0/3262164509 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5f7fe700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf600fbcd0 0x7faf60066d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 -- 172.21.15.100:0/3262164509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf600fbcd0 msgr2=0x7faf60066d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf600fbcd0 0x7faf60066d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 -- 172.21.15.100:0/3262164509 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf54005040 con 0x7faf600fcf80 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf5effd700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf60112710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faf54009150 tx=0x7faf54009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf5400e040 con 0x7faf600fcf80 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf60113010 con 0x7faf600fcf80 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7faf5400bcc0 con 0x7faf600fcf80 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7faf5401c630 con 0x7faf600fcf80 2024-03-24T11:17:18.651 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.648+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf601135d0 con 0x7faf600fcf80 2024-03-24T11:17:18.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.649+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf6004c710 con 0x7faf600fcf80 2024-03-24T11:17:18.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.650+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7faf54013420 con 0x7faf600fcf80 2024-03-24T11:17:18.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.650+0000 7faf47fff700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf480692a0 0x7faf4806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:18.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.650+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7faf54087f90 con 0x7faf600fcf80 2024-03-24T11:17:18.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.650+0000 7faf5f7fe700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf480692a0 0x7faf4806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:18.652 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.651+0000 7faf5f7fe700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf480692a0 0x7faf4806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faf50007fd0 tx=0x7faf50009040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:18.655 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.655+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7faf54057d50 con 0x7faf600fcf80 2024-03-24T11:17:18.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:18 smithi105 ceph-mon[36437]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:17:18.856 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.856+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}] v 0) v1 -- 0x7faf601139b0 con 0x7faf600fcf80 2024-03-24T11:17:18.857 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:18 smithi100 ceph-mon[28941]: pgmap v196: 65 pgs: 65 active+clean; 579 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 938 B/s rd, 170 B/s wr, 1 op/s 2024-03-24T11:17:18.857 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.856+0000 7faf47fff700 1 -- 172.21.15.100:0/3262164509 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim}]=0 v21)=0 v21) v1 ==== 155+0+0 (secure 0 0 0) 0x7faf540578e0 con 0x7faf600fcf80 2024-03-24T11:17:18.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf480692a0 msgr2=0x7faf4806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:18.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf480692a0 0x7faf4806b750 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7faf50007fd0 tx=0x7faf50009040 comp rx=0 tx=0).stop 2024-03-24T11:17:18.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 msgr2=0x7faf60112710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:18.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf60112710 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7faf54009150 tx=0x7faf54009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:18.859 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 shutdown_connections 2024-03-24T11:17:18.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7faf480692a0 0x7faf4806b750 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:18.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf600fbcd0 0x7faf60066d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:18.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 --2- 172.21.15.100:0/3262164509 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7faf600fcf80 0x7faf60112710 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:18.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 >> 172.21.15.100:0/3262164509 conn(0x7faf600f7430 msgr2=0x7faf600f8b50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:18.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 shutdown_connections 2024-03-24T11:17:18.860 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:18.858+0000 7faf65ca4700 1 -- 172.21.15.100:0/3262164509 wait complete. 2024-03-24T11:17:19.799 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force' 2024-03-24T11:17:20.107 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:20.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:20 smithi100 ceph-mon[28941]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-24T11:17:20.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:20 smithi105 ceph-mon[36437]: pgmap v197: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 -- 172.21.15.100:0/87946429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 msgr2=0x7f1f800fe840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 --2- 172.21.15.100:0/87946429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f800fe840 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f1f70004940 tx=0x7f1f70009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 -- 172.21.15.100:0/87946429 shutdown_connections 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 --2- 172.21.15.100:0/87946429 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f800fed80 0x7f1f800faec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 --2- 172.21.15.100:0/87946429 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f800fe840 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 -- 172.21.15.100:0/87946429 >> 172.21.15.100:0/87946429 conn(0x7f1f800f6bd0 msgr2=0x7f1f800f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 -- 172.21.15.100:0/87946429 shutdown_connections 2024-03-24T11:17:21.724 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 -- 172.21.15.100:0/87946429 wait complete. 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.722+0000 7f1f85a04700 1 Processor -- start 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f85a04700 1 -- start start 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f85a04700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f8010d920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f85a04700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f800fed80 0x7f1f8010de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f85a04700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f8010e4f0 con 0x7f1f800fe430 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f85a04700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1f80112e90 con 0x7f1f800fed80 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f8010d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f8010d920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53842/0 (socket says 172.21.15.100:53842) 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 -- 172.21.15.100:0/732538735 learned_addr learned my addr 172.21.15.100:0/732538735 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f77fff700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f800fed80 0x7f1f8010de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 -- 172.21.15.100:0/732538735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f800fed80 msgr2=0x7f1f8010de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f800fed80 0x7f1f8010de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 -- 172.21.15.100:0/732538735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1f70005040 con 0x7f1f800fe430 2024-03-24T11:17:21.725 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f7effd700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f8010d920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000b3b0 tx=0x7f1f70009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:21.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f7000e040 con 0x7f1f800fe430 2024-03-24T11:17:21.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f1f7000bbf0 con 0x7f1f800fe430 2024-03-24T11:17:21.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.723+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1f80113090 con 0x7f1f800fe430 2024-03-24T11:17:21.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.724+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1f7001c630 con 0x7f1f800fe430 2024-03-24T11:17:21.726 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.724+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1f80113600 con 0x7f1f800fe430 2024-03-24T11:17:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.724+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1f8004c710 con 0x7f1f800fe430 2024-03-24T11:17:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.725+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f1f7000bd60 con 0x7f1f800fe430 2024-03-24T11:17:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.725+0000 7f1f777fe700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f1f6c0692a0 0x7f1f6c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.725+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f1f70087e40 con 0x7f1f800fe430 2024-03-24T11:17:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.725+0000 7f1f77fff700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f1f6c0692a0 0x7f1f6c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:21.727 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.726+0000 7f1f77fff700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f1f6c0692a0 0x7f1f6c06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009bf0 tx=0x7f1f680091a0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:21.730 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.730+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f1f70057c00 con 0x7f1f800fe430 2024-03-24T11:17:21.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.915+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}] v 0) v1 -- 0x7f1f80113960 con 0x7f1f800fe430 2024-03-24T11:17:21.915 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.915+0000 7f1f777fe700 1 -- 172.21.15.100:0/732538735 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_warn_on_insecure_global_id_reclaim_allowed}]=0 v21)=0 v21) v1 ==== 163+0+0 (secure 0 0 0) 0x7f1f70057790 con 0x7f1f800fe430 2024-03-24T11:17:21.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f1f6c0692a0 msgr2=0x7f1f6c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:21.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f1f6c0692a0 0x7f1f6c06b750 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f1f68009bf0 tx=0x7f1f680091a0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.917 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 msgr2=0x7f1f8010d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f8010d920 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f1f7000b3b0 tx=0x7f1f70009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 shutdown_connections 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f1f6c0692a0 0x7f1f6c06b750 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f1f800fe430 0x7f1f8010d920 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 --2- 172.21.15.100:0/732538735 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1f800fed80 0x7f1f8010de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 >> 172.21.15.100:0/732538735 conn(0x7f1f800f6bd0 msgr2=0x7f1f800f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 shutdown_connections 2024-03-24T11:17:21.918 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:21.917+0000 7f1f85a04700 1 -- 172.21.15.100:0/732538735 wait complete. 2024-03-24T11:17:22.588 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:22 smithi100 ceph-mon[28941]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-24T11:17:22.616 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph config set global log_to_journald false --force' 2024-03-24T11:17:22.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:22 smithi105 ceph-mon[36437]: pgmap v198: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.4 KiB/s rd, 682 B/s wr, 2 op/s 2024-03-24T11:17:22.873 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:23.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:23.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:24.539 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.537+0000 7f4d867e2700 1 -- 172.21.15.100:0/3012636131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d80066ef0 msgr2=0x7f4d800ffa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.537+0000 7f4d867e2700 1 --2- 172.21.15.100:0/3012636131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d80066ef0 0x7f4d800ffa40 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f4d70004940 tx=0x7f4d70009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.537+0000 7f4d867e2700 1 -- 172.21.15.100:0/3012636131 shutdown_connections 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.537+0000 7f4d867e2700 1 --2- 172.21.15.100:0/3012636131 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d800fff80 0x7f4d80102360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.537+0000 7f4d867e2700 1 --2- 172.21.15.100:0/3012636131 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d80066ef0 0x7f4d800ffa40 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.537+0000 7f4d867e2700 1 -- 172.21.15.100:0/3012636131 >> 172.21.15.100:0/3012636131 conn(0x7f4d800f7430 msgr2=0x7f4d800f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 -- 172.21.15.100:0/3012636131 shutdown_connections 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 -- 172.21.15.100:0/3012636131 wait complete. 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 Processor -- start 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 -- start start 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d80066ef0 0x7f4d8010d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 0x7f4d8010de90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:24.540 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d8010e520 con 0x7f4d800fff80 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d867e2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d80112ec0 con 0x7f4d80066ef0 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.538+0000 7f4d7f7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 0x7f4d8010de90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7f7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 0x7f4d8010de90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53870/0 (socket says 172.21.15.100:53870) 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7f7fe700 1 -- 172.21.15.100:0/2477846871 learned_addr learned my addr 172.21.15.100:0/2477846871 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7ffff700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d80066ef0 0x7f4d8010d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7f7fe700 1 -- 172.21.15.100:0/2477846871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d80066ef0 msgr2=0x7f4d8010d950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7f7fe700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d80066ef0 0x7f4d8010d950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7f7fe700 1 -- 172.21.15.100:0/2477846871 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d70005040 con 0x7f4d800fff80 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7f7fe700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 0x7f4d8010de90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400b790 tx=0x7f4d74009250 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d7400f6f0 con 0x7f4d800fff80 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d80113120 con 0x7f4d800fff80 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f4d7400fd20 con 0x7f4d800fff80 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d740103a0 con 0x7f4d800fff80 2024-03-24T11:17:24.541 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.539+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d80113690 con 0x7f4d800fff80 2024-03-24T11:17:24.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.540+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d8004c710 con 0x7f4d800fff80 2024-03-24T11:17:24.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.541+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f4d74011400 con 0x7f4d800fff80 2024-03-24T11:17:24.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.541+0000 7f4d7cff9700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4d680692a0 0x7f4d6806b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:24.542 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.541+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f4d74086f70 con 0x7f4d800fff80 2024-03-24T11:17:24.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.541+0000 7f4d7ffff700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4d680692a0 0x7f4d6806b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:24.543 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.542+0000 7f4d7ffff700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4d680692a0 0x7f4d6806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d70000c00 tx=0x7f4d70009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:24.546 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.546+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f4d74056d30 con 0x7f4d800fff80 2024-03-24T11:17:24.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:24 smithi100 ceph-mon[28941]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-24T11:17:24.735 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.734+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command([{prefix=config set, name=log_to_journald}] v 0) v1 -- 0x7f4d80105930 con 0x7f4d800fff80 2024-03-24T11:17:24.736 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.735+0000 7f4d7cff9700 1 -- 172.21.15.100:0/2477846871 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{prefix=config set, name=log_to_journald}]=0 v21)=0 v21) v1 ==== 135+0+0 (secure 0 0 0) 0x7f4d740568c0 con 0x7f4d800fff80 2024-03-24T11:17:24.737 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4d680692a0 msgr2=0x7f4d6806b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:24.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4d680692a0 0x7f4d6806b750 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f4d70000c00 tx=0x7f4d70009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:24.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 msgr2=0x7f4d8010de90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:24.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 0x7f4d8010de90 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7f4d7400b790 tx=0x7f4d74009250 comp rx=0 tx=0).stop 2024-03-24T11:17:24.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 shutdown_connections 2024-03-24T11:17:24.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f4d680692a0 0x7f4d6806b750 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:24.738 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d80066ef0 0x7f4d8010d950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:24.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 --2- 172.21.15.100:0/2477846871 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f4d800fff80 0x7f4d8010de90 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:24.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 >> 172.21.15.100:0/2477846871 conn(0x7f4d800f7430 msgr2=0x7f4d800f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:24.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 shutdown_connections 2024-03-24T11:17:24.739 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:24.737+0000 7f4d867e2700 1 -- 172.21.15.100:0/2477846871 wait complete. 2024-03-24T11:17:24.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:24 smithi105 ceph-mon[36437]: pgmap v199: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.8 KiB/s rd, 682 B/s wr, 3 op/s 2024-03-24T11:17:25.503 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 --daemon-types mgr' 2024-03-24T11:17:25.758 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:17:26.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:26 smithi105 ceph-mon[36437]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:17:26.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:26 smithi100 ceph-mon[28941]: pgmap v200: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 -- 172.21.15.100:0/2974970371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 msgr2=0x7f397c102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 --2- 172.21.15.100:0/2974970371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c102360 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7f3970004940 tx=0x7f3970009e30 comp rx=0 tx=0).stop 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 -- 172.21.15.100:0/2974970371 shutdown_connections 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 --2- 172.21.15.100:0/2974970371 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c102360 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 --2- 172.21.15.100:0/2974970371 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f397c066ed0 0x7f397c0ffa40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 -- 172.21.15.100:0/2974970371 >> 172.21.15.100:0/2974970371 conn(0x7f397c0f7450 msgr2=0x7f397c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:17:27.512 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 -- 172.21.15.100:0/2974970371 shutdown_connections 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.510+0000 7f39826d7700 1 -- 172.21.15.100:0/2974970371 wait complete. 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f39826d7700 1 Processor -- start 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f39826d7700 1 -- start start 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f39826d7700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f397c066ed0 0x7f397c0fd290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f39826d7700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c0fb940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f39826d7700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0fd890 con 0x7f397c0fff80 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f39826d7700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f397c0fbe80 con 0x7f397c066ed0 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f397b7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c0fb940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f397b7fe700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c0fb940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:53882/0 (socket says 172.21.15.100:53882) 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.511+0000 7f397b7fe700 1 -- 172.21.15.100:0/2198701332 learned_addr learned my addr 172.21.15.100:0/2198701332 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f397bfff700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f397c066ed0 0x7f397c0fd290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f397b7fe700 1 -- 172.21.15.100:0/2198701332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f397c066ed0 msgr2=0x7f397c0fd290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f397b7fe700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f397c066ed0 0x7f397c0fd290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f397b7fe700 1 -- 172.21.15.100:0/2198701332 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3970005040 con 0x7f397c0fff80 2024-03-24T11:17:27.513 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f397b7fe700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c0fb940 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f397000b3b0 tx=0x7f3970009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f397000e040 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f397000bbf0 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f397c0fc0d0 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f397001c630 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.512+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f397c0fc690 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.513+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f397c04c710 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.514+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f397000bd60 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.514+0000 7f3978ff9700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f39640692a0 0x7f396406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.514+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f3970087d70 con 0x7f397c0fff80 2024-03-24T11:17:27.515 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.514+0000 7f397bfff700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f39640692a0 0x7f396406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:17:27.516 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.515+0000 7f397bfff700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f39640692a0 0x7f396406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f397c066c10 tx=0x7f396c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:17:27.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.519+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f3970057bd0 con 0x7f397c0fff80 2024-03-24T11:17:27.723 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:17:27.723+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "daemon_types": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f397c0f8f40 con 0x7f39640692a0 2024-03-24T11:17:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:28 smithi105 ceph-mon[36437]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T11:17:28.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:28 smithi105 ceph-mon[36437]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:28.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:28 smithi100 ceph-mon[28941]: pgmap v201: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.3 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T11:17:28.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:28 smithi100 ceph-mon[28941]: from='client.14556 -' entity='client.admin' cmd=[{"prefix": "orch upgrade start", "image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", "daemon_types": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:17:30.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:30 smithi105 ceph-mon[36437]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T11:17:30.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:30 smithi100 ceph-mon[28941]: pgmap v202: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.7 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T11:17:32.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:32 smithi105 ceph-mon[36437]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:17:32.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:32 smithi100 ceph-mon[28941]: pgmap v203: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:17:34.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:34 smithi105 ceph-mon[36437]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-24T11:17:34.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:34 smithi100 ceph-mon[28941]: pgmap v204: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-24T11:17:36.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:36 smithi105 ceph-mon[36437]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:17:36.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:36 smithi100 ceph-mon[28941]: pgmap v205: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:17:38.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:38 smithi105 ceph-mon[36437]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:17:38.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:38.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:38 smithi100 ceph-mon[28941]: pgmap v206: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:17:38.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:40.796 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:40 smithi105 ceph-mon[36437]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:40.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:40 smithi100 ceph-mon[28941]: pgmap v207: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:42.802 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:42 smithi105 ceph-mon[36437]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:42.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:42 smithi100 ceph-mon[28941]: pgmap v208: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:44.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:43 smithi100 ceph-mon[28941]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:44.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:43 smithi105 ceph-mon[36437]: pgmap v209: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:46 smithi105 ceph-mon[36437]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:46.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:46 smithi100 ceph-mon[28941]: pgmap v210: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:48 smithi105 ceph-mon[36437]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:48.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:48 smithi100 ceph-mon[28941]: pgmap v211: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:50 smithi105 ceph-mon[36437]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:50.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:50 smithi100 ceph-mon[28941]: pgmap v212: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:17:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:52 smithi105 ceph-mon[36437]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:52.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:52 smithi100 ceph-mon[28941]: pgmap v213: 65 pgs: 65 active+clean; 584 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:17:53.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:53.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:17:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:54 smithi105 ceph-mon[36437]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T11:17:54.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:54 smithi100 ceph-mon[28941]: pgmap v214: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T11:17:56.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:56 smithi105 ceph-mon[36437]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T11:17:56.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:56 smithi100 ceph-mon[28941]: pgmap v215: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 426 B/s wr, 2 op/s 2024-03-24T11:17:58.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:17:58 smithi105 ceph-mon[36437]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:17:58.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:17:58 smithi100 ceph-mon[28941]: pgmap v216: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 511 B/s wr, 2 op/s 2024-03-24T11:18:00.555 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:00 smithi105 ceph-mon[36437]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T11:18:00.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:00 smithi100 ceph-mon[28941]: pgmap v217: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 511 B/s wr, 3 op/s 2024-03-24T11:18:02.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:02 smithi100 ceph-mon[28941]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T11:18:02.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:02 smithi105 ceph-mon[36437]: pgmap v218: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 597 B/s wr, 2 op/s 2024-03-24T11:18:03.094 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.094+0000 7f3978ff9700 1 -- 172.21.15.100:0/2198701332 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+89 (secure 0 0 0) 0x7f397c0f8f40 con 0x7f39640692a0 2024-03-24T11:18:03.095 INFO:teuthology.orchestra.run.smithi100.stdout:Initiating upgrade to quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f39640692a0 msgr2=0x7f396406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f39640692a0 0x7f396406b750 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f397c066c10 tx=0x7f396c00b040 comp rx=0 tx=0).stop 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 msgr2=0x7f397c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c0fb940 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f397000b3b0 tx=0x7f3970009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 shutdown_connections 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f39640692a0 0x7f396406b750 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f397c066ed0 0x7f397c0fd290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 --2- 172.21.15.100:0/2198701332 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f397c0fff80 0x7f397c0fb940 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 >> 172.21.15.100:0/2198701332 conn(0x7f397c0f7450 msgr2=0x7f397c0f7820 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 shutdown_connections 2024-03-24T11:18:03.097 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:03.096+0000 7f39826d7700 1 -- 172.21.15.100:0/2198701332 wait complete. 2024-03-24T11:18:03.737 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'while ceph orch upgrade status | jq '"'"'.in_progress'"'"' | grep true && ! ceph orch upgrade status | jq '"'"'.message'"'"' | grep Error ; do ceph orch ps ; ceph versions ; ceph orch upgrade status ; sleep 30 ; done' 2024-03-24T11:18:04.108 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:04.356 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:04 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:04.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: Upgrade: Started with target quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:18:04.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:04.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:04.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: pgmap v219: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.6 KiB/s rd, 597 B/s wr, 3 op/s 2024-03-24T11:18:04.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:04.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-03-24T11:18:04.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:04 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:05.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:05 smithi105 ceph-mon[36437]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:18:05.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:05 smithi100 ceph-mon[28941]: Upgrade: First pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 2024-03-24T11:18:06.332 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:06 smithi100 ceph-mon[28941]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-03-24T11:18:06.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:06 smithi105 ceph-mon[36437]: pgmap v220: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-03-24T11:18:08.527 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:08 smithi100 ceph-mon[28941]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:08.527 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:08 smithi100 ceph-mon[28941]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-03-24T11:18:08.527 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:08 smithi100 ceph-mon[28941]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:08.527 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:08 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:08.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:08 smithi105 ceph-mon[36437]: from='client.14560 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:08.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:08 smithi105 ceph-mon[36437]: pgmap v221: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 170 B/s wr, 2 op/s 2024-03-24T11:18:08.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:08 smithi105 ceph-mon[36437]: from='client.14564 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:08.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:08 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:09.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:09 smithi105 ceph-mon[36437]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:09.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:09 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/3848899042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:18:09.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:09 smithi100 ceph-mon[28941]: from='client.24319 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:09.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:09 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/3848899042' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:18:10.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:10 smithi105 ceph-mon[36437]: from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:10.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:10 smithi105 ceph-mon[36437]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-24T11:18:10.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:10 smithi100 ceph-mon[28941]: from='client.24327 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:10.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:10 smithi100 ceph-mon[28941]: pgmap v222: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 85 B/s wr, 2 op/s 2024-03-24T11:18:12.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:12 smithi105 ceph-mon[36437]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:18:12.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:12 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:12.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:12 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:12.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:12 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:12.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:12 smithi100 ceph-mon[28941]: pgmap v223: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 85 B/s wr, 1 op/s 2024-03-24T11:18:12.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:12 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:12.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:12 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:12.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:12 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 -- 172.21.15.100:0/922577976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 msgr2=0x7f8fa40faf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/922577976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40faf30 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f8f94004940 tx=0x7f8f94009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 -- 172.21.15.100:0/922577976 shutdown_connections 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/922577976 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40faf30 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/922577976 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fa40fe4a0 0x7f8fa40fe8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 -- 172.21.15.100:0/922577976 >> 172.21.15.100:0/922577976 conn(0x7f8fa40f6ba0 msgr2=0x7f8fa40f8fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.766 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 -- 172.21.15.100:0/922577976 shutdown_connections 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.530+0000 7f8fab5e2700 1 -- 172.21.15.100:0/922577976 wait complete. 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fab5e2700 1 Processor -- start 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fab5e2700 1 -- start start 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fab5e2700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fa40fe4a0 0x7f8fa40fdcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fab5e2700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40fc1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fab5e2700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4111420 con 0x7f8fa40fedf0 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fab5e2700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fa4111590 con 0x7f8fa40fe4a0 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fa937e700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fa40fe4a0 0x7f8fa40fdcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fa8b7d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40fc1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fa8b7d700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40fc1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:58130/0 (socket says 172.21.15.100:58130) 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.531+0000 7f8fa8b7d700 1 -- 172.21.15.100:0/487579989 learned_addr learned my addr 172.21.15.100:0/487579989 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8fa8b7d700 1 -- 172.21.15.100:0/487579989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fa40fe4a0 msgr2=0x7f8fa40fdcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8fa8b7d700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fa40fe4a0 0x7f8fa40fdcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.767 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8fa8b7d700 1 -- 172.21.15.100:0/487579989 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f94005040 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8fa8b7d700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40fc1c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8f940043d0 tx=0x7f8f94009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f94020070 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f8f94013d00 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.532+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fa40fc7c0 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.533+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fa40fcbe0 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.534+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8f9400e070 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.535+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f8f94025410 con 0x7f8fa40fedf0 2024-03-24T11:18:12.768 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-03-24T11:18:12.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.535+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f880052e0 con 0x7f8fa40fedf0 2024-03-24T11:18:12.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.535+0000 7f8f99ffb700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f8f90069490 0x7f8f9006b940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.769 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.535+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f8f94089630 con 0x7f8fa40fedf0 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.535+0000 7f8fa937e700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f8f90069490 0x7f8f9006b940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.536+0000 7f8fa937e700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f8f90069490 0x7f8f9006b940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009fd0 tx=0x7f8fa000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.544+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f8f940593d0 con 0x7f8fa40fedf0 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.838+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f8f88000bf0 con 0x7f8f90069490 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.839+0000 7f8f99ffb700 1 -- 172.21.15.100:0/487579989 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7f8f88000bf0 con 0x7f8f90069490 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.841+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f8f90069490 msgr2=0x7f8f9006b940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.841+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f8f90069490 0x7f8f9006b940 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f8fa0009fd0 tx=0x7f8fa000b040 comp rx=0 tx=0).stop 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 msgr2=0x7f8fa40fc1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40fc1c0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f8f940043d0 tx=0x7f8f94009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 shutdown_connections 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f8f90069490 0x7f8f9006b940 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8fa40fe4a0 0x7f8fa40fdcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 --2- 172.21.15.100:0/487579989 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f8fa40fedf0 0x7f8fa40fc1c0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.770 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 >> 172.21.15.100:0/487579989 conn(0x7f8fa40f6ba0 msgr2=0x7f8fa40f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 shutdown_connections 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.842+0000 7f8fab5e2700 1 -- 172.21.15.100:0/487579989 wait complete. 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 -- 172.21.15.100:0/359611869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 msgr2=0x7ff6740fc0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/359611869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 0x7ff6740fc0e0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7ff670004940 tx=0x7ff670009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 -- 172.21.15.100:0/359611869 shutdown_connections 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/359611869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff6740fd3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/359611869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 0x7ff6740fc0e0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 -- 172.21.15.100:0/359611869 >> 172.21.15.100:0/359611869 conn(0x7ff6740f7430 msgr2=0x7ff6740f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 -- 172.21.15.100:0/359611869 shutdown_connections 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 -- 172.21.15.100:0/359611869 wait complete. 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.988+0000 7ff67b4c0700 1 Processor -- start 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff67b4c0700 1 -- start start 2024-03-24T11:18:12.771 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff67b4c0700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 0x7ff6741106b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff67b4c0700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff674112c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff67b4c0700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674066350 con 0x7ff6740fcf80 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff67b4c0700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff674066490 con 0x7ff6740fbcd0 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff678a5b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff674112c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff678a5b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff674112c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:58152/0 (socket says 172.21.15.100:58152) 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff678a5b700 1 -- 172.21.15.100:0/1304104869 learned_addr learned my addr 172.21.15.100:0/1304104869 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff67925c700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 0x7ff6741106b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff678a5b700 1 -- 172.21.15.100:0/1304104869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 msgr2=0x7ff6741106b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff678a5b700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 0x7ff6741106b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.989+0000 7ff678a5b700 1 -- 172.21.15.100:0/1304104869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff664005040 con 0x7ff6740fcf80 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.990+0000 7ff678a5b700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff674112c00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff6640096d0 tx=0x7ff664009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.990+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff66400e840 con 0x7ff6740fcf80 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.990+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7ff66400f040 con 0x7ff6740fcf80 2024-03-24T11:18:12.772 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.990+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff670005040 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.990+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff67410c8c0 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.990+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff664013dc0 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.992+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff67405f5f0 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.992+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7ff66401c410 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.992+0000 7ff669ffb700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff6600692a0 0x7ff66006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.992+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7ff664089030 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.993+0000 7ff67925c700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff6600692a0 0x7ff66006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.994+0000 7ff67925c700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff6600692a0 0x7ff66006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff670013870 tx=0x7ff67001d040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:06.997+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7ff664052990 con 0x7ff6740fcf80 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.230+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7ff674101220 con 0x7ff6600692a0 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.232+0000 7ff669ffb700 1 -- 172.21.15.100:0/1304104869 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7ff674101220 con 0x7ff6600692a0 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.234+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff6600692a0 msgr2=0x7ff66006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.234+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff6600692a0 0x7ff66006b750 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7ff670013870 tx=0x7ff67001d040 comp rx=0 tx=0).stop 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.234+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 msgr2=0x7ff674112c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.773 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.234+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff674112c00 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff6640096d0 tx=0x7ff664009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.235+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 shutdown_connections 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.235+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7ff6600692a0 0x7ff66006b750 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.235+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff6740fbcd0 0x7ff6741106b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.235+0000 7ff67b4c0700 1 --2- 172.21.15.100:0/1304104869 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7ff6740fcf80 0x7ff674112c00 secure :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7ff6640096d0 tx=0x7ff664009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.235+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 >> 172.21.15.100:0/1304104869 conn(0x7ff6740f7430 msgr2=0x7ff6740f8a80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.236+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 shutdown_connections 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.236+0000 7ff67b4c0700 1 -- 172.21.15.100:0/1304104869 wait complete. 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.396+0000 7fdc58da9700 1 -- 172.21.15.100:0/1695584536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 msgr2=0x7fdc540faed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.396+0000 7fdc58da9700 1 --2- 172.21.15.100:0/1695584536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc540faed0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7fdc48004940 tx=0x7fdc48009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.396+0000 7fdc58da9700 1 -- 172.21.15.100:0/1695584536 shutdown_connections 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.396+0000 7fdc58da9700 1 --2- 172.21.15.100:0/1695584536 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc540faed0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.396+0000 7fdc58da9700 1 --2- 172.21.15.100:0/1695584536 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc540fe850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.396+0000 7fdc58da9700 1 -- 172.21.15.100:0/1695584536 >> 172.21.15.100:0/1695584536 conn(0x7fdc540f6bc0 msgr2=0x7fdc540f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.397+0000 7fdc58da9700 1 -- 172.21.15.100:0/1695584536 shutdown_connections 2024-03-24T11:18:12.774 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.397+0000 7fdc58da9700 1 -- 172.21.15.100:0/1695584536 wait complete. 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.398+0000 7fdc58da9700 1 Processor -- start 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.398+0000 7fdc58da9700 1 -- start start 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.398+0000 7fdc58da9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc5410d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.398+0000 7fdc58da9700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc5410ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.398+0000 7fdc5259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc5410d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.398+0000 7fdc5259c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc5410d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.100:49074/0 (socket says 172.21.15.100:49074) 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc51d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc5410ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc51d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc5410ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:58172/0 (socket says 172.21.15.100:58172) 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc5259c700 1 -- 172.21.15.100:0/2100196733 learned_addr learned my addr 172.21.15.100:0/2100196733 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5410e3f0 con 0x7fdc540fed90 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc5410e560 con 0x7fdc540fe440 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc5259c700 1 -- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 msgr2=0x7fdc5410ddf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc5259c700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc5410ddf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.775 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc5259c700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc48005040 con 0x7fdc540fe440 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.399+0000 7fdc51d9b700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc5410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.400+0000 7fdc5259c700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc5410d8b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400eb40 tx=0x7fdc4400b040 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.400+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc44007cc0 con 0x7fdc540fe440 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.400+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc54112f60 con 0x7fdc540fe440 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.400+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc54113410 con 0x7fdc540fe440 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.400+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fdc4400f040 con 0x7fdc540fe440 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.smithi100 smithi100 *:9093,9094 running (5m) 61s ago 6m 25.0M - 0.25.0 c8568f914cd2 1c0d9b7ea6a4 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stdout:crash.smithi100 smithi100 running (7m) 61s ago 7m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 6221438d109b 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stdout:crash.smithi105 smithi105 running (6m) 68s ago 6m 6656k - 17.2.7-726-g9c4e0c34 837124da804f 6007c22dfa6c 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.smithi100 smithi100 *:3000 running (5m) 61s ago 6m 82.4M - 9.4.7 954c08fa6188 884de3e3cb33 2024-03-24T11:18:12.776 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi100.begydc smithi100 running (87s) 61s ago 86s 16.4M - 17.2.7-726-g9c4e0c34 837124da804f 13c54dffbff4 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi100.ccifff smithi100 running (81s) 61s ago 80s 12.8M - 17.2.7-726-g9c4e0c34 837124da804f 35eee9e6be58 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi105.kkjmwl smithi105 running (84s) 68s ago 83s 10.4M - 17.2.7-726-g9c4e0c34 837124da804f e27ce1402b71 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi105.rwkhto smithi105 running (78s) 68s ago 78s 11.9M - 17.2.7-726-g9c4e0c34 837124da804f c841622d4951 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.smithi100.smxgyi smithi100 *:9283 running (8m) 61s ago 8m 481M - 17.2.7-726-g9c4e0c34 837124da804f 3a17926ffe53 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.smithi105.nmspmy smithi105 *:8443,9283 running (5m) 68s ago 5m 433M - 17.2.7-726-g9c4e0c34 837124da804f 20760d28e4ea 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mon.smithi100 smithi100 running (8m) 61s ago 8m 45.1M 2048M 17.2.7-726-g9c4e0c34 837124da804f f7fbf127ce8a 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:mon.smithi105 smithi105 running (5m) 68s ago 5m 37.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 243836c787a4 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.smithi100 smithi100 *:9100 running (6m) 61s ago 6m 15.7M - 1.5.0 0da6a335fe13 71e6d8a3aef2 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.smithi105 smithi105 *:9100 running (6m) 68s ago 5m 15.8M - 1.5.0 0da6a335fe13 8fd41687715a 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (5m) 61s ago 5m 46.7M 4288M 17.2.7-726-g9c4e0c34 837124da804f d8dbc515015a 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (4m) 61s ago 4m 48.0M 4288M 17.2.7-726-g9c4e0c34 837124da804f 8c2b1a04b4bf 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (4m) 61s ago 4m 44.0M 4288M 17.2.7-726-g9c4e0c34 837124da804f e04c18a8a009 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi105 running (3m) 68s ago 3m 47.1M 5312M 17.2.7-726-g9c4e0c34 837124da804f 97b4dd2a4b6e 2024-03-24T11:18:12.777 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi105 running (3m) 68s ago 3m 44.4M 5312M 17.2.7-726-g9c4e0c34 837124da804f 1f71a14b0353 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi105 running (2m) 68s ago 2m 41.9M 5312M 17.2.7-726-g9c4e0c34 837124da804f 211e50c6219d 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.smithi100 smithi100 *:9095 running (5m) 61s ago 6m 56.9M - 2.43.0 a07b618ecd1d 6a8a1455db93 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.400+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc4400ca00 con 0x7fdc540fe440 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.402+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fdc4400cb60 con 0x7fdc540fe440 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.402+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc5404c710 con 0x7fdc540fe440 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.403+0000 7fdc42ffd700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fdc3c0692a0 0x7fdc3c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.403+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fdc44087c60 con 0x7fdc540fe440 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.403+0000 7fdc51d9b700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fdc3c0692a0 0x7fdc3c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.404+0000 7fdc51d9b700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fdc3c0692a0 0x7fdc3c06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc48009b30 tx=0x7fdc48009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.411+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fdc4404e190 con 0x7fdc540fe440 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.811+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7fdc54113680 con 0x7fdc3c0692a0 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.821+0000 7fdc42ffd700 1 -- 172.21.15.100:0/2100196733 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3366 (secure 0 0 0) 0x7fdc54113680 con 0x7fdc3c0692a0 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.823+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fdc3c0692a0 msgr2=0x7fdc3c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.823+0000 7fdc58da9700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fdc3c0692a0 0x7fdc3c06b750 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7fdc48009b30 tx=0x7fdc48009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.823+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 msgr2=0x7fdc5410d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.823+0000 7fdc58da9700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc5410d8b0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7fdc4400eb40 tx=0x7fdc4400b040 comp rx=0 tx=0).stop 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 shutdown_connections 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fdc3c0692a0 0x7fdc3c06b750 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc540fe440 0x7fdc5410d8b0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 --2- 172.21.15.100:0/2100196733 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fdc540fed90 0x7fdc5410ddf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 >> 172.21.15.100:0/2100196733 conn(0x7fdc540f6bc0 msgr2=0x7fdc540f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 shutdown_connections 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.824+0000 7fdc58da9700 1 -- 172.21.15.100:0/2100196733 wait complete. 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e49ffb700 1 -- 172.21.15.100:0/3492437125 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4000bcb0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 -- 172.21.15.100:0/3492437125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 msgr2=0x7f5e4c0fe540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3492437125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c0fe540 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f5e40004940 tx=0x7f5e40009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 -- 172.21.15.100:0/3492437125 shutdown_connections 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3492437125 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c0fe540 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3492437125 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e4c0fcee0 0x7f5e4c0fd310 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 -- 172.21.15.100:0/3492437125 >> 172.21.15.100:0/3492437125 conn(0x7f5e4c0f8560 msgr2=0x7f5e4c0fa970 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 -- 172.21.15.100:0/3492437125 shutdown_connections 2024-03-24T11:18:12.780 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.929+0000 7f5e515b9700 1 -- 172.21.15.100:0/3492437125 wait complete. 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.930+0000 7f5e515b9700 1 Processor -- start 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.930+0000 7f5e515b9700 1 -- start start 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.930+0000 7f5e515b9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e4c0fcee0 0x7f5e4c113e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.930+0000 7f5e515b9700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c1143a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.930+0000 7f5e515b9700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c115110 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.930+0000 7f5e515b9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e4c115280 con 0x7f5e4c0fcee0 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4affd700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e4c0fcee0 0x7f5e4c113e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c1143a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c1143a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:57046/0 (socket says 172.21.15.100:57046) 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 -- 172.21.15.100:0/3848899042 learned_addr learned my addr 172.21.15.100:0/3848899042 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 -- 172.21.15.100:0/3848899042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e4c0fcee0 msgr2=0x7f5e4c113e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e4c0fcee0 0x7f5e4c113e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.781 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 -- 172.21.15.100:0/3848899042 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e40005040 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-24T11:18:12.782 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": { 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 14 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e4a7fc700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c1143a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5e40000c00 tx=0x7f5e40009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4000e040 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e4c1154a0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.783 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.931+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e4c1159e0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.932+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f5e400138f0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.932+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5e4001c630 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.933+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e380052e0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.933+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f5e4001c8a0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.934+0000 7f5e337fe700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5e340692a0 0x7f5e3406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.934+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f5e400885e0 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.934+0000 7f5e4affd700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5e340692a0 0x7f5e3406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.934+0000 7f5e4affd700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5e340692a0 0x7f5e3406b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c113bd0 tx=0x7f5e3c007ea0 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:07.939+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f5e40054a20 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.408+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f5e38005d00 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.409+0000 7f5e337fe700 1 -- 172.21.15.100:0/3848899042 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f5e40058040 con 0x7f5e4c0fe0d0 2024-03-24T11:18:12.784 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.411+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5e340692a0 msgr2=0x7f5e3406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.411+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5e340692a0 0x7f5e3406b750 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f5e4c113bd0 tx=0x7f5e3c007ea0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.411+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 msgr2=0x7f5e4c1143a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.411+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c1143a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f5e40000c00 tx=0x7f5e40009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 shutdown_connections 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69", 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-03-24T11:18:12.785 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "", 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Doing first pull of quay.ceph.io/ceph-ci/ceph:db0330b1e4e2470d52b750e251e55a522b4f7d69 image", 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f5e340692a0 0x7f5e3406b750 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5e4c0fcee0 0x7f5e4c113e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 --2- 172.21.15.100:0/3848899042 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f5e4c0fe0d0 0x7f5e4c1143a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 >> 172.21.15.100:0/3848899042 conn(0x7f5e4c0f8560 msgr2=0x7f5e4c0f98f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 shutdown_connections 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.412+0000 7f5e515b9700 1 -- 172.21.15.100:0/3848899042 wait complete. 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.526+0000 7fd570893700 1 -- 172.21.15.100:0/1290524050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fcf80 msgr2=0x7fd56c0fd3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.526+0000 7fd570893700 1 --2- 172.21.15.100:0/1290524050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fcf80 0x7fd56c0fd3f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7fd560004940 tx=0x7fd560009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.526+0000 7fd570893700 1 -- 172.21.15.100:0/1290524050 shutdown_connections 2024-03-24T11:18:12.786 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.526+0000 7fd570893700 1 --2- 172.21.15.100:0/1290524050 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fcf80 0x7fd56c0fd3f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.526+0000 7fd570893700 1 --2- 172.21.15.100:0/1290524050 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fbcd0 0x7fd56c0fc0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.527+0000 7fd570893700 1 -- 172.21.15.100:0/1290524050 >> 172.21.15.100:0/1290524050 conn(0x7fd56c0f7430 msgr2=0x7fd56c0f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.527+0000 7fd570893700 1 -- 172.21.15.100:0/1290524050 shutdown_connections 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.527+0000 7fd570893700 1 -- 172.21.15.100:0/1290524050 wait complete. 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd570893700 1 Processor -- start 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd570893700 1 -- start start 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd570893700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fbcd0 0x7fd56c10e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd570893700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 0x7fd56c1109c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd570893700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c066370 con 0x7fd56c0fbcd0 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd570893700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56c0664b0 con 0x7fd56c0fcf80 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd569d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 0x7fd56c1109c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd569d9b700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 0x7fd56c1109c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.100:59132/0 (socket says 172.21.15.100:59132) 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd569d9b700 1 -- 172.21.15.100:0/2864470894 learned_addr learned my addr 172.21.15.100:0/2864470894 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:12.787 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.528+0000 7fd56a59c700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fbcd0 0x7fd56c10e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd569d9b700 1 -- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fbcd0 msgr2=0x7fd56c10e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd569d9b700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fbcd0 0x7fd56c10e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd569d9b700 1 -- 172.21.15.100:0/2864470894 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560005040 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd56a59c700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fbcd0 0x7fd56c10e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd569d9b700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 0x7fd56c1109c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd560013870 tx=0x7fd560009e30 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56000e070 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7fd56001c400 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd56c110f60 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd56000b5f0 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.529+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56c111380 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.530+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56c05f5f0 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.531+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7fd56001ca50 con 0x7fd56c0fcf80 2024-03-24T11:18:12.788 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.532+0000 7fd55affd700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd5540692b0 0x7fd55406b760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.532+0000 7fd56a59c700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd5540692b0 0x7fd55406b760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.532+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7fd560088810 con 0x7fd56c0fcf80 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.532+0000 7fd56a59c700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd5540692b0 0x7fd55406b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd56c065ff0 tx=0x7fd55c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.540+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7fd560054be0 con 0x7fd56c0fcf80 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.952+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7fd56c1012a0 con 0x7fd5540692b0 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.954+0000 7fd55affd700 1 -- 172.21.15.100:0/2864470894 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+351 (secure 0 0 0) 0x7fd56c1012a0 con 0x7fd5540692b0 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.956+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd5540692b0 msgr2=0x7fd55406b760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.956+0000 7fd570893700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd5540692b0 0x7fd55406b760 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fd56c065ff0 tx=0x7fd55c00b040 comp rx=0 tx=0).stop 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.956+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 msgr2=0x7fd56c1109c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.956+0000 7fd570893700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 0x7fd56c1109c0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fd560013870 tx=0x7fd560009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 shutdown_connections 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7fd5540692b0 0x7fd55406b760 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7fd56c0fbcd0 0x7fd56c10e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.789 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 --2- 172.21.15.100:0/2864470894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd56c0fcf80 0x7fd56c1109c0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:12.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 >> 172.21.15.100:0/2864470894 conn(0x7fd56c0f7430 msgr2=0x7fd56c0f8ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:12.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 shutdown_connections 2024-03-24T11:18:12.790 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:08.957+0000 7fd570893700 1 -- 172.21.15.100:0/2864470894 wait complete. 2024-03-24T11:18:13.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:13 smithi105 ceph-mon[36437]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:18:13.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:13 smithi105 ceph-mon[36437]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:18:13.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:13 smithi105 ceph-mon[36437]: Upgrade: Need to upgrade myself (mgr.smithi100.smxgyi) 2024-03-24T11:18:13.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:13 smithi105 ceph-mon[36437]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 on smithi105 2024-03-24T11:18:13.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:13 smithi100 ceph-mon[28941]: Upgrade: Target is version 19.0.0-1667-gdb0330b1 (unknown) 2024-03-24T11:18:13.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:13 smithi100 ceph-mon[28941]: Upgrade: Target container is quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10, digests ['quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10'] 2024-03-24T11:18:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:13 smithi100 ceph-mon[28941]: Upgrade: Need to upgrade myself (mgr.smithi100.smxgyi) 2024-03-24T11:18:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:13 smithi100 ceph-mon[28941]: Upgrade: Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 on smithi105 2024-03-24T11:18:14.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:14 smithi105 ceph-mon[36437]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:18:14.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:14 smithi100 ceph-mon[28941]: pgmap v224: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:18:16.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:16 smithi105 ceph-mon[36437]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:16.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:16 smithi100 ceph-mon[28941]: pgmap v225: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:18.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:18 smithi105 ceph-mon[36437]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:18.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:18 smithi100 ceph-mon[28941]: pgmap v226: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:20.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:20 smithi105 ceph-mon[36437]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:18:20.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:20 smithi100 ceph-mon[28941]: pgmap v227: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:18:22.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:22 smithi105 ceph-mon[36437]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:22.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:22 smithi100 ceph-mon[28941]: pgmap v228: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:23.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:23 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:23.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:23 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:24.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:24 smithi105 ceph-mon[36437]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:18:24.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:24 smithi100 ceph-mon[28941]: pgmap v229: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 1.2 KiB/s rd, 2 op/s 2024-03-24T11:18:25.260 INFO:tasks.workunit.client.1.smithi105.stderr:Updating files: 92% (11496/12399) Updating files: 93% (11532/12399) Updating files: 94% (11656/12399) Updating files: 95% (11780/12399) Updating files: 96% (11904/12399) Updating files: 97% (12028/12399) Updating files: 98% (12152/12399) Updating files: 99% (12276/12399) Updating files: 100% (12399/12399) Updating files: 100% (12399/12399), done. 2024-03-24T11:18:25.699 INFO:tasks.workunit.client.1.smithi105.stderr:Note: switching to 'f73f1dfd96849300c4322bd7259a3ab37f94c200'. 2024-03-24T11:18:25.699 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.699 INFO:tasks.workunit.client.1.smithi105.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T11:18:25.699 INFO:tasks.workunit.client.1.smithi105.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T11:18:25.699 INFO:tasks.workunit.client.1.smithi105.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: git switch -c 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr:Or undo this operation with: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: git switch - 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr: 2024-03-24T11:18:25.700 INFO:tasks.workunit.client.1.smithi105.stderr:HEAD is now at f73f1dfd968 Merge pull request #56388 from rhcs-dashboard/wip-64960-squid 2024-03-24T11:18:25.704 DEBUG:teuthology.orchestra.run.smithi105:> cd -- /home/ubuntu/cephtest/clone.client.1/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.1 2024-03-24T11:18:25.765 INFO:tasks.workunit.client.1.smithi105.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T11:18:25.767 INFO:tasks.workunit.client.1.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-24T11:18:25.767 INFO:tasks.workunit.client.1.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:Note: switching to 'f73f1dfd96849300c4322bd7259a3ab37f94c200'. 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:changes and commit them, and you can discard any commits you make in this 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:state without impacting any branches by switching back to a branch. 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:If you want to create a new branch to retain commits you create, you may 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:do so (now or later) by using -c with the switch command. Example: 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr: git switch -c 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.990 INFO:tasks.workunit.client.0.smithi100.stderr:Or undo this operation with: 2024-03-24T11:18:25.991 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.991 INFO:tasks.workunit.client.0.smithi100.stderr: git switch - 2024-03-24T11:18:25.991 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.991 INFO:tasks.workunit.client.0.smithi100.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-03-24T11:18:25.991 INFO:tasks.workunit.client.0.smithi100.stderr: 2024-03-24T11:18:25.991 INFO:tasks.workunit.client.0.smithi100.stderr:HEAD is now at f73f1dfd968 Merge pull request #56388 from rhcs-dashboard/wip-64960-squid 2024-03-24T11:18:25.994 DEBUG:teuthology.orchestra.run.smithi100:> cd -- /home/ubuntu/cephtest/clone.client.0/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.client.0 2024-03-24T11:18:26.123 INFO:tasks.workunit.client.0.smithi100.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-03-24T11:18:26.125 INFO:tasks.workunit.client.0.smithi100.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T11:18:26.125 INFO:tasks.workunit.client.0.smithi100.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-03-24T11:18:26.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:26 smithi105 ceph-mon[36437]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:26.578 INFO:tasks.workunit.client.1.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T11:18:26.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:26 smithi100 ceph-mon[28941]: pgmap v230: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:26.643 INFO:tasks.workunit.client.1.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T11:18:26.688 INFO:tasks.workunit.client.1.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/direct_io' 2024-03-24T11:18:26.689 INFO:tasks.workunit.client.1.smithi105.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-24T11:18:26.689 INFO:tasks.workunit.client.1.smithi105.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T11:18:26.730 INFO:tasks.workunit.client.1.smithi105.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.1/qa/workunits/fs' 2024-03-24T11:18:26.733 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:18:26.733 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/workunits.list.client.1 of=/dev/stdout 2024-03-24T11:18:26.789 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.1... 2024-03-24T11:18:26.790 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-24T11:18:26.791 DEBUG:teuthology.orchestra.run.smithi105:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && cd -- /home/ubuntu/cephtest/mnt.1/client.1/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f73f1dfd96849300c4322bd7259a3ab37f94c200 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="1" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.1 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.1 CEPH_MNT=/home/ubuntu/cephtest/mnt.1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.1/qa/workunits/suites/fsstress.sh 2024-03-24T11:18:26.850 INFO:tasks.workunit.client.1.smithi105.stderr:+ mkdir -p fsstress 2024-03-24T11:18:26.852 INFO:tasks.workunit.client.1.smithi105.stderr:+ pushd fsstress 2024-03-24T11:18:26.852 INFO:tasks.workunit.client.1.smithi105.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-24T11:18:26.852 INFO:tasks.workunit.client.1.smithi105.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-24T11:18:26.872 INFO:tasks.workunit.client.0.smithi100.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-03-24T11:18:26.948 INFO:tasks.workunit.client.0.smithi100.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-03-24T11:18:26.988 INFO:tasks.workunit.client.0.smithi100.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/direct_io' 2024-03-24T11:18:26.989 INFO:tasks.workunit.client.0.smithi100.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T11:18:26.990 INFO:tasks.workunit.client.0.smithi100.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-03-24T11:18:27.029 INFO:tasks.workunit.client.0.smithi100.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.client.0/qa/workunits/fs' 2024-03-24T11:18:27.032 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:18:27.033 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/home/ubuntu/cephtest/workunits.list.client.0 of=/dev/stdout 2024-03-24T11:18:27.048 INFO:tasks.workunit:Running workunits matching suites/fsstress.sh on client.0... 2024-03-24T11:18:27.049 INFO:tasks.workunit:Running workunit suites/fsstress.sh... 2024-03-24T11:18:27.050 DEBUG:teuthology.orchestra.run.smithi100:workunit test suites/fsstress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && cd -- /home/ubuntu/cephtest/mnt.0/client.0/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=f73f1dfd96849300c4322bd7259a3ab37f94c200 TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster ceph" CEPH_ID="0" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.client.0 CEPH_ROOT=/home/ubuntu/cephtest/clone.client.0 CEPH_MNT=/home/ubuntu/cephtest/mnt.0 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 3h /home/ubuntu/cephtest/clone.client.0/qa/workunits/suites/fsstress.sh 2024-03-24T11:18:27.107 INFO:tasks.workunit.client.0.smithi100.stderr:+ mkdir -p fsstress 2024-03-24T11:18:27.109 INFO:tasks.workunit.client.0.smithi100.stderr:+ pushd fsstress 2024-03-24T11:18:27.109 INFO:tasks.workunit.client.0.smithi100.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-24T11:18:27.109 INFO:tasks.workunit.client.0.smithi100.stderr:+ wget -q -O ltp-full.tgz http://download.ceph.com/qa/ltp-full-20091231.tgz 2024-03-24T11:18:27.404 INFO:tasks.workunit.client.1.smithi105.stderr:+ tar xzf ltp-full.tgz 2024-03-24T11:18:28.349 INFO:tasks.workunit.client.0.smithi100.stderr:+ tar xzf ltp-full.tgz 2024-03-24T11:18:28.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:28 smithi105 ceph-mon[36437]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:28.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:28 smithi100 ceph-mon[28941]: pgmap v231: 65 pgs: 65 active+clean; 587 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 853 B/s rd, 1 op/s 2024-03-24T11:18:30.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:30 smithi105 ceph-mon[36437]: pgmap v232: 65 pgs: 65 active+clean; 1.7 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 98 KiB/s wr, 2 op/s 2024-03-24T11:18:30.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:30 smithi100 ceph-mon[28941]: pgmap v232: 65 pgs: 65 active+clean; 1.7 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 2.7 KiB/s rd, 98 KiB/s wr, 2 op/s 2024-03-24T11:18:32.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:32 smithi105 ceph-mon[36437]: pgmap v233: 65 pgs: 65 active+clean; 5.7 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 344 KiB/s rd, 439 KiB/s wr, 2 op/s 2024-03-24T11:18:32.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:32 smithi100 ceph-mon[28941]: pgmap v233: 65 pgs: 65 active+clean; 5.7 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 344 KiB/s rd, 439 KiB/s wr, 2 op/s 2024-03-24T11:18:34.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:34 smithi105 ceph-mon[36437]: pgmap v234: 65 pgs: 65 active+clean; 19 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 970 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-03-24T11:18:34.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:34 smithi100 ceph-mon[28941]: pgmap v234: 65 pgs: 65 active+clean; 19 MiB data, 1.7 GiB used, 535 GiB / 536 GiB avail; 970 KiB/s rd, 1.6 MiB/s wr, 78 op/s 2024-03-24T11:18:36.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:36 smithi105 ceph-mon[36437]: pgmap v235: 65 pgs: 65 active+clean; 36 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 970 KiB/s rd, 3.1 MiB/s wr, 255 op/s 2024-03-24T11:18:36.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:36 smithi100 ceph-mon[28941]: pgmap v235: 65 pgs: 65 active+clean; 36 MiB data, 1.8 GiB used, 535 GiB / 536 GiB avail; 970 KiB/s rd, 3.1 MiB/s wr, 255 op/s 2024-03-24T11:18:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:38 smithi105 ceph-mon[36437]: pgmap v236: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 474 op/s 2024-03-24T11:18:38.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:38 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:38.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:38 smithi100 ceph-mon[28941]: pgmap v236: 65 pgs: 65 active+clean; 62 MiB data, 1.9 GiB used, 534 GiB / 536 GiB avail; 1.3 MiB/s rd, 5.4 MiB/s wr, 474 op/s 2024-03-24T11:18:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:38 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:39.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 -- 172.21.15.100:0/181184330 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 msgr2=0x7f60fc0fd410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.103 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 --2- 172.21.15.100:0/181184330 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc0fd410 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7f60f0004940 tx=0x7f60f0009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 -- 172.21.15.100:0/181184330 shutdown_connections 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 --2- 172.21.15.100:0/181184330 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc0fd410 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 --2- 172.21.15.100:0/181184330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc0fc240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 -- 172.21.15.100:0/181184330 >> 172.21.15.100:0/181184330 conn(0x7f60fc0f7480 msgr2=0x7f60fc0f9890 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 -- 172.21.15.100:0/181184330 shutdown_connections 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.101+0000 7f6100b93700 1 -- 172.21.15.100:0/181184330 wait complete. 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f6100b93700 1 Processor -- start 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f6100b93700 1 -- start start 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f6100b93700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc1106f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f6100b93700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc112c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f6100b93700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc066b90 con 0x7f60fc0fcfa0 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f6100b93700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc066cd0 con 0x7f60fc0fbe30 2024-03-24T11:18:39.104 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60f9d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc112c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60fa59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc1106f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60f9d9b700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc112c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43742/0 (socket says 172.21.15.100:43742) 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60fa59c700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc1106f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.100:59108/0 (socket says 172.21.15.100:59108) 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60f9d9b700 1 -- 172.21.15.100:0/3497826612 learned_addr learned my addr 172.21.15.100:0/3497826612 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60fa59c700 1 -- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 msgr2=0x7f60fc112c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60fa59c700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc112c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60fa59c700 1 -- 172.21.15.100:0/3497826612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60f0005040 con 0x7f60fc0fbe30 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.102+0000 7f60f9d9b700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.103+0000 7f60fa59c700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc1106f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f60ec002a50 tx=0x7f60ec004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.103+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60ec007cc0 con 0x7f60fc0fbe30 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.103+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc10c530 con 0x7f60fc0fbe30 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.103+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f60ec00f040 con 0x7f60fc0fbe30 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.103+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f60ec009530 con 0x7f60fc0fbe30 2024-03-24T11:18:39.105 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.103+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc10caa0 con 0x7f60fc0fbe30 2024-03-24T11:18:39.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.104+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc05f5f0 con 0x7f60fc0fbe30 2024-03-24T11:18:39.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.104+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f60ec0097f0 con 0x7f60fc0fbe30 2024-03-24T11:18:39.106 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.105+0000 7f60eaffd700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f60e40692a0 0x7f60e406b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.105+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f60ec086f80 con 0x7f60fc0fbe30 2024-03-24T11:18:39.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.105+0000 7f60f9d9b700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f60e40692a0 0x7f60e406b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.107 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.106+0000 7f60f9d9b700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f60e40692a0 0x7f60e406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f60f0004f90 tx=0x7f60f0009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:39.110 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.110+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f60ec053190 con 0x7f60fc0fbe30 2024-03-24T11:18:39.396 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.396+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f60fc101270 con 0x7f60e40692a0 2024-03-24T11:18:39.397 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.397+0000 7f60eaffd700 1 -- 172.21.15.100:0/3497826612 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f60fc101270 con 0x7f60e40692a0 2024-03-24T11:18:39.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f60e40692a0 msgr2=0x7f60e406b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f60e40692a0 0x7f60e406b750 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f60f0004f90 tx=0x7f60f0009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:39.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 msgr2=0x7f60fc1106f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc1106f0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7f60ec002a50 tx=0x7f60ec004600 comp rx=0 tx=0).stop 2024-03-24T11:18:39.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 shutdown_connections 2024-03-24T11:18:39.400 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f60e40692a0 0x7f60e406b750 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f60fc0fbe30 0x7f60fc1106f0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 --2- 172.21.15.100:0/3497826612 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f60fc0fcfa0 0x7f60fc112c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 >> 172.21.15.100:0/3497826612 conn(0x7f60fc0f7480 msgr2=0x7f60fc0f8ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:39.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 shutdown_connections 2024-03-24T11:18:39.401 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.400+0000 7f6100b93700 1 -- 172.21.15.100:0/3497826612 wait complete. 2024-03-24T11:18:39.410 INFO:teuthology.orchestra.run.smithi100.stdout:true 2024-03-24T11:18:39.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 -- 172.21.15.100:0/2222295493 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 msgr2=0x7f12080fd300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 --2- 172.21.15.100:0/2222295493 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f12080fd300 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7f11f8004940 tx=0x7f11f8009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:39.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 -- 172.21.15.100:0/2222295493 shutdown_connections 2024-03-24T11:18:39.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 --2- 172.21.15.100:0/2222295493 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 0x7f12080fe4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 --2- 172.21.15.100:0/2222295493 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f12080fd300 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.518 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 -- 172.21.15.100:0/2222295493 >> 172.21.15.100:0/2222295493 conn(0x7f12080f85b0 msgr2=0x7f12080fa9e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:39.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 -- 172.21.15.100:0/2222295493 shutdown_connections 2024-03-24T11:18:39.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.517+0000 7f120e6c9700 1 -- 172.21.15.100:0/2222295493 wait complete. 2024-03-24T11:18:39.519 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f120e6c9700 1 Processor -- start 2024-03-24T11:18:39.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f120e6c9700 1 -- start start 2024-03-24T11:18:39.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f120e6c9700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f120810d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f120e6c9700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 0x7f120810def0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f120e6c9700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120810e4f0 con 0x7f12080fcef0 2024-03-24T11:18:39.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f120e6c9700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f120810ed40 con 0x7f12080fe060 2024-03-24T11:18:39.520 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f1207fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f120810d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f1207fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f120810d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43750/0 (socket says 172.21.15.100:43750) 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.518+0000 7f1207fff700 1 -- 172.21.15.100:0/3054496344 learned_addr learned my addr 172.21.15.100:0/3054496344 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f12077fe700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 0x7f120810def0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f12077fe700 1 -- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 msgr2=0x7f120810d9b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f12077fe700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f120810d9b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f12077fe700 1 -- 172.21.15.100:0/3054496344 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f11f8005040 con 0x7f12080fe060 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f1207fff700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f120810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f12077fe700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 0x7f120810def0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f11fc002a50 tx=0x7f11fc004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mon.1 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11fc007cc0 con 0x7f12080fe060 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mon.1 v2:172.21.15.105:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f11fc00f040 con 0x7f12080fe060 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f120810f020 con 0x7f12080fe060 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mon.1 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f11fc0094b0 con 0x7f12080fe060 2024-03-24T11:18:39.521 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.519+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f120810f560 con 0x7f12080fe060 2024-03-24T11:18:39.522 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.520+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f120805f5f0 con 0x7f12080fe060 2024-03-24T11:18:39.522 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.521+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mon.1 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f11fc009770 con 0x7f12080fe060 2024-03-24T11:18:39.522 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.521+0000 7f1204ff9700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f11f00692a0 0x7f11f006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.521+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mon.1 v2:172.21.15.105:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f11fc086ff0 con 0x7f12080fe060 2024-03-24T11:18:39.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.521+0000 7f1207fff700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f11f00692a0 0x7f11f006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.523 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.522+0000 7f1207fff700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f11f00692a0 0x7f11f006b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11f8004f90 tx=0x7f11f8009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:39.527 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.526+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mon.1 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f11fc056cb0 con 0x7f12080fe060 2024-03-24T11:18:39.838 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.837+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f120810f8f0 con 0x7f11f00692a0 2024-03-24T11:18:39.839 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.839+0000 7f1204ff9700 1 -- 172.21.15.100:0/3054496344 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f120810f8f0 con 0x7f11f00692a0 2024-03-24T11:18:39.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f11f00692a0 msgr2=0x7f11f006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f11f00692a0 0x7f11f006b750 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f11f8004f90 tx=0x7f11f8009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:39.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 msgr2=0x7f120810def0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.841 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 0x7f120810def0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f11fc002a50 tx=0x7f11fc004600 comp rx=0 tx=0).stop 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 shutdown_connections 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f11f00692a0 0x7f11f006b750 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f12080fcef0 0x7f120810d9b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 --2- 172.21.15.100:0/3054496344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f12080fe060 0x7f120810def0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 >> 172.21.15.100:0/3054496344 conn(0x7f12080f85b0 msgr2=0x7f12080f9c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 shutdown_connections 2024-03-24T11:18:39.842 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.841+0000 7f120e6c9700 1 -- 172.21.15.100:0/3054496344 wait complete. 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 -- 172.21.15.100:0/3022388491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 msgr2=0x7f71600ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 --2- 172.21.15.100:0/3022388491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f71600ff9c0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7f715c004940 tx=0x7f715c009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 -- 172.21.15.100:0/3022388491 shutdown_connections 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 --2- 172.21.15.100:0/3022388491 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71600fff00 0x7f7160102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 --2- 172.21.15.100:0/3022388491 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f71600ff9c0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 -- 172.21.15.100:0/3022388491 >> 172.21.15.100:0/3022388491 conn(0x7f71600f7430 msgr2=0x7f71600f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 -- 172.21.15.100:0/3022388491 shutdown_connections 2024-03-24T11:18:39.960 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.959+0000 7f71680bc700 1 -- 172.21.15.100:0/3022388491 wait complete. 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f71680bc700 1 Processor -- start 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f71680bc700 1 -- start start 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f71680bc700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f716010da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f71680bc700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71600fff00 0x7f716010df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f71680bc700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716010e570 con 0x7f71600662e0 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f71680bc700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f716010edc0 con 0x7f71600fff00 2024-03-24T11:18:39.962 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f7165e58700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f716010da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f7165e58700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f716010da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43774/0 (socket says 172.21.15.100:43774) 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f7165657700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71600fff00 0x7f716010df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.960+0000 7f7165e58700 1 -- 172.21.15.100:0/4028604730 learned_addr learned my addr 172.21.15.100:0/4028604730 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f7165e58700 1 -- 172.21.15.100:0/4028604730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71600fff00 msgr2=0x7f716010df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f7165e58700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71600fff00 0x7f716010df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f7165e58700 1 -- 172.21.15.100:0/4028604730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f715c005040 con 0x7f71600662e0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f7165e58700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f716010da30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f715c00b3b0 tx=0x7f715c009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f715c00e040 con 0x7f71600662e0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f715c00bbf0 con 0x7f71600662e0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f715c01c630 con 0x7f71600662e0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.961+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f716010f040 con 0x7f71600662e0 2024-03-24T11:18:39.963 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.962+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f716010f580 con 0x7f71600662e0 2024-03-24T11:18:39.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.963+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f716005f5f0 con 0x7f71600662e0 2024-03-24T11:18:39.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.964+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f715c00bd60 con 0x7f71600662e0 2024-03-24T11:18:39.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.964+0000 7f71567fc700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f714c0692a0 0x7f714c06b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:39.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.964+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f715c087da0 con 0x7f71600662e0 2024-03-24T11:18:39.964 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.964+0000 7f7165657700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f714c0692a0 0x7f714c06b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:39.965 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.965+0000 7f7165657700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f714c0692a0 0x7f714c06b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7150004f40 tx=0x7f715000b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:39.969 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:39.969+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f715c057c00 con 0x7f71600662e0 2024-03-24T11:18:40.252 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.252+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch ps", "target": ["mon-mgr", ""]}) v1 -- 0x7f716010f900 con 0x7f714c0692a0 2024-03-24T11:18:40.270 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.270+0000 7f71567fc700 1 -- 172.21.15.100:0/4028604730 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+3410 (secure 0 0 0) 0x7f716010f900 con 0x7f714c0692a0 2024-03-24T11:18:40.270 INFO:teuthology.orchestra.run.smithi100.stdout:NAME HOST PORTS STATUS REFRESHED AGE MEM USE MEM LIM VERSION IMAGE ID CONTAINER ID 2024-03-24T11:18:40.270 INFO:teuthology.orchestra.run.smithi100.stdout:alertmanager.smithi100 smithi100 *:9093,9094 running (6m) 93s ago 7m 25.0M - 0.25.0 c8568f914cd2 1c0d9b7ea6a4 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:crash.smithi100 smithi100 running (7m) 93s ago 7m 6660k - 17.2.7-726-g9c4e0c34 837124da804f 6221438d109b 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:crash.smithi105 smithi105 running (6m) 100s ago 6m 6656k - 17.2.7-726-g9c4e0c34 837124da804f 6007c22dfa6c 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:grafana.smithi100 smithi100 *:3000 running (5m) 93s ago 7m 82.4M - 9.4.7 954c08fa6188 884de3e3cb33 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi100.begydc smithi100 running (119s) 93s ago 118s 16.4M - 17.2.7-726-g9c4e0c34 837124da804f 13c54dffbff4 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi100.ccifff smithi100 running (113s) 93s ago 113s 12.8M - 17.2.7-726-g9c4e0c34 837124da804f 35eee9e6be58 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi105.kkjmwl smithi105 running (116s) 100s ago 116s 10.4M - 17.2.7-726-g9c4e0c34 837124da804f e27ce1402b71 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mds.cephfs.smithi105.rwkhto smithi105 running (111s) 100s ago 110s 11.9M - 17.2.7-726-g9c4e0c34 837124da804f c841622d4951 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.smithi100.smxgyi smithi100 *:9283 running (8m) 93s ago 8m 481M - 17.2.7-726-g9c4e0c34 837124da804f 3a17926ffe53 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mgr.smithi105.nmspmy smithi105 *:8443,9283 running (6m) 100s ago 6m 433M - 17.2.7-726-g9c4e0c34 837124da804f 20760d28e4ea 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mon.smithi100 smithi100 running (8m) 93s ago 8m 45.1M 2048M 17.2.7-726-g9c4e0c34 837124da804f f7fbf127ce8a 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:mon.smithi105 smithi105 running (6m) 100s ago 6m 37.3M 2048M 17.2.7-726-g9c4e0c34 837124da804f 243836c787a4 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.smithi100 smithi100 *:9100 running (7m) 93s ago 7m 15.7M - 1.5.0 0da6a335fe13 71e6d8a3aef2 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:node-exporter.smithi105 smithi105 *:9100 running (6m) 100s ago 6m 15.8M - 1.5.0 0da6a335fe13 8fd41687715a 2024-03-24T11:18:40.271 INFO:teuthology.orchestra.run.smithi100.stdout:osd.0 smithi100 running (5m) 93s ago 5m 46.7M 4288M 17.2.7-726-g9c4e0c34 837124da804f d8dbc515015a 2024-03-24T11:18:40.272 INFO:teuthology.orchestra.run.smithi100.stdout:osd.1 smithi100 running (5m) 93s ago 5m 48.0M 4288M 17.2.7-726-g9c4e0c34 837124da804f 8c2b1a04b4bf 2024-03-24T11:18:40.272 INFO:teuthology.orchestra.run.smithi100.stdout:osd.2 smithi100 running (4m) 93s ago 4m 44.0M 4288M 17.2.7-726-g9c4e0c34 837124da804f e04c18a8a009 2024-03-24T11:18:40.272 INFO:teuthology.orchestra.run.smithi100.stdout:osd.3 smithi105 running (4m) 100s ago 4m 47.1M 5312M 17.2.7-726-g9c4e0c34 837124da804f 97b4dd2a4b6e 2024-03-24T11:18:40.272 INFO:teuthology.orchestra.run.smithi100.stdout:osd.4 smithi105 running (3m) 100s ago 3m 44.4M 5312M 17.2.7-726-g9c4e0c34 837124da804f 1f71a14b0353 2024-03-24T11:18:40.272 INFO:teuthology.orchestra.run.smithi100.stdout:osd.5 smithi105 running (3m) 100s ago 3m 41.9M 5312M 17.2.7-726-g9c4e0c34 837124da804f 211e50c6219d 2024-03-24T11:18:40.272 INFO:teuthology.orchestra.run.smithi100.stdout:prometheus.smithi100 smithi100 *:9095 running (5m) 93s ago 6m 56.9M - 2.43.0 a07b618ecd1d 6a8a1455db93 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f714c0692a0 msgr2=0x7f714c06b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f714c0692a0 0x7f714c06b750 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f7150004f40 tx=0x7f715000b040 comp rx=0 tx=0).stop 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 msgr2=0x7f716010da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f716010da30 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7f715c00b3b0 tx=0x7f715c009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 shutdown_connections 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f714c0692a0 0x7f714c06b750 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f71600662e0 0x7f716010da30 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.273 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 --2- 172.21.15.100:0/4028604730 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71600fff00 0x7f716010df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.272+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 >> 172.21.15.100:0/4028604730 conn(0x7f71600f7430 msgr2=0x7f71600f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:40.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.273+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 shutdown_connections 2024-03-24T11:18:40.274 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.273+0000 7f71680bc700 1 -- 172.21.15.100:0/4028604730 wait complete. 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 -- 172.21.15.100:0/2743083669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fcc60 msgr2=0x7f58d80fd0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 --2- 172.21.15.100:0/2743083669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fcc60 0x7f58d80fd0d0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7f58cc004940 tx=0x7f58cc009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 -- 172.21.15.100:0/2743083669 shutdown_connections 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 --2- 172.21.15.100:0/2743083669 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fcc60 0x7f58d80fd0d0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 --2- 172.21.15.100:0/2743083669 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58d80fc310 0x7f58d80fc720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 -- 172.21.15.100:0/2743083669 >> 172.21.15.100:0/2743083669 conn(0x7f58d80f6be0 msgr2=0x7f58d80f9010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 -- 172.21.15.100:0/2743083669 shutdown_connections 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.401+0000 7f58de765700 1 -- 172.21.15.100:0/2743083669 wait complete. 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58de765700 1 Processor -- start 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58de765700 1 -- start start 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58de765700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 0x7f58d80fba10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:40.403 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58de765700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58d80fcc60 0x7f58d81126c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:40.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58de765700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d8112c00 con 0x7f58d80fc310 2024-03-24T11:18:40.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58de765700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58d8112d40 con 0x7f58d80fcc60 2024-03-24T11:18:40.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58d7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 0x7f58d80fba10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:40.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58d7fff700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 0x7f58d80fba10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43796/0 (socket says 172.21.15.100:43796) 2024-03-24T11:18:40.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58d7fff700 1 -- 172.21.15.100:0/405013375 learned_addr learned my addr 172.21.15.100:0/405013375 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:40.404 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.402+0000 7f58d77fe700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58d80fcc60 0x7f58d81126c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d7fff700 1 -- 172.21.15.100:0/405013375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58d80fcc60 msgr2=0x7f58d81126c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d7fff700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58d80fcc60 0x7f58d81126c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d7fff700 1 -- 172.21.15.100:0/405013375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58cc005040 con 0x7f58d80fc310 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d7fff700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 0x7f58d80fba10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58c800eb70 tx=0x7f58c800b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58c8007cc0 con 0x7f58d80fc310 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f58c800f040 con 0x7f58d80fc310 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58d8112f40 con 0x7f58d80fc310 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58c800ca90 con 0x7f58d80fc310 2024-03-24T11:18:40.405 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.403+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58d81133f0 con 0x7f58d80fc310 2024-03-24T11:18:40.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.404+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58c40052e0 con 0x7f58d80fc310 2024-03-24T11:18:40.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.405+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f58c8009710 con 0x7f58d80fc310 2024-03-24T11:18:40.406 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.406+0000 7f58d4ff9700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f58c00692a0 0x7f58c006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:40.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.406+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f58c8086c00 con 0x7f58d80fc310 2024-03-24T11:18:40.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.406+0000 7f58d77fe700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f58c00692a0 0x7f58c006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:40.407 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.406+0000 7f58d77fe700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f58c00692a0 0x7f58c006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009b30 tx=0x7f58cc009e30 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:40.410 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.410+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f58c808c020 con 0x7f58d80fc310 2024-03-24T11:18:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:40 smithi105 ceph-mon[36437]: pgmap v237: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 507 op/s 2024-03-24T11:18:40.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:40 smithi105 ceph-mon[36437]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:40.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:40 smithi100 ceph-mon[28941]: pgmap v237: 65 pgs: 65 active+clean; 71 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.0 MiB/s rd, 6.2 MiB/s wr, 507 op/s 2024-03-24T11:18:40.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:40 smithi100 ceph-mon[28941]: from='client.24331 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:40.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.775+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "versions"} v 0) v1 -- 0x7f58c4005da0 con 0x7f58d80fc310 2024-03-24T11:18:40.776 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.776+0000 7f58d4ff9700 1 -- 172.21.15.100:0/405013375 <== mon.0 v2:172.21.15.100:3300/0 7 ==== mon_command_ack([{"prefix": "versions"}]=0 v0) v1 ==== 56+0+638 (secure 0 0 0) 0x7f58c80569c0 con 0x7f58d80fc310 2024-03-24T11:18:40.776 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-03-24T11:18:40.776 INFO:teuthology.orchestra.run.smithi100.stdout: "mon": { 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "mgr": { 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 2 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "osd": { 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 6 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "mds": { 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 4 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: }, 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "overall": { 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: "ceph version 17.2.7-726-g9c4e0c34 (9c4e0c34f1949e7d705fe7ff26a4f4b729df7a45) quincy (stable)": 14 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout: } 2024-03-24T11:18:40.777 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-03-24T11:18:40.778 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f58c00692a0 msgr2=0x7f58c006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f58c00692a0 0x7f58c006b750 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f58cc009b30 tx=0x7f58cc009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 msgr2=0x7f58d80fba10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 0x7f58d80fba10 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f58c800eb70 tx=0x7f58c800b040 comp rx=0 tx=0).stop 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 shutdown_connections 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f58c00692a0 0x7f58c006b750 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f58d80fc310 0x7f58d80fba10 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 --2- 172.21.15.100:0/405013375 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58d80fcc60 0x7f58d81126c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 >> 172.21.15.100:0/405013375 conn(0x7f58d80f6be0 msgr2=0x7f58d80f8200 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 shutdown_connections 2024-03-24T11:18:40.779 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.778+0000 7f58de765700 1 -- 172.21.15.100:0/405013375 wait complete. 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 -- 172.21.15.100:0/673466364 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 msgr2=0x7f0e780ff9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/673466364 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e780ff9c0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f0e68004940 tx=0x7f0e68009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 -- 172.21.15.100:0/673466364 shutdown_connections 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/673466364 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e780fff00 0x7f0e78102320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/673466364 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e780ff9c0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 -- 172.21.15.100:0/673466364 >> 172.21.15.100:0/673466364 conn(0x7f0e780f7430 msgr2=0x7f0e780f9860 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 -- 172.21.15.100:0/673466364 shutdown_connections 2024-03-24T11:18:40.889 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.888+0000 7f0e7cccb700 1 -- 172.21.15.100:0/673466364 wait complete. 2024-03-24T11:18:40.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7cccb700 1 Processor -- start 2024-03-24T11:18:40.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7cccb700 1 -- start start 2024-03-24T11:18:40.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7cccb700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e7810da30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:40.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7cccb700 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e780fff00 0x7f0e7810df70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:40.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7cccb700 1 -- --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7810e570 con 0x7f0e780662e0 2024-03-24T11:18:40.891 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7cccb700 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e7810edc0 con 0x7f0e780fff00 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e7810da30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7659c700 1 --2- >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e7810da30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.100:3300/0 says I am v2:172.21.15.100:43816/0 (socket says 172.21.15.100:43816) 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e7659c700 1 -- 172.21.15.100:0/2961944783 learned_addr learned my addr 172.21.15.100:0/2961944783 (peer_addr_for_me v2:172.21.15.100:0/0) 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.889+0000 7f0e75d9b700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e780fff00 0x7f0e7810df70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e7659c700 1 -- 172.21.15.100:0/2961944783 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e780fff00 msgr2=0x7f0e7810df70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e7659c700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e780fff00 0x7f0e7810df70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e7659c700 1 -- 172.21.15.100:0/2961944783 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e68005040 con 0x7f0e780662e0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e7659c700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e7810da30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e68000c00 tx=0x7f0e68009e30 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mon.0 v2:172.21.15.100:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e6800e040 con 0x7f0e780662e0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mon.0 v2:172.21.15.100:3300/0 2 ==== config(27 keys) v1 ==== 1109+0+0 (secure 0 0 0) 0x7f0e6800bbf0 con 0x7f0e780662e0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e7810f040 con 0x7f0e780662e0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mon.0 v2:172.21.15.100:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0e6801c630 con 0x7f0e780662e0 2024-03-24T11:18:40.892 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.890+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e7810f580 con 0x7f0e780662e0 2024-03-24T11:18:40.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.891+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 --> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e7805f5f0 con 0x7f0e780662e0 2024-03-24T11:18:40.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.892+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mon.0 v2:172.21.15.100:3300/0 4 ==== mgrmap(e 19) v1 ==== 86623+0+0 (secure 0 0 0) 0x7f0e6800bd60 con 0x7f0e780662e0 2024-03-24T11:18:40.893 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.892+0000 7f0e66ffd700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0e600692a0 0x7f0e6006b750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-03-24T11:18:40.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.892+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mon.0 v2:172.21.15.100:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5464+0+0 (secure 0 0 0) 0x7f0e68088480 con 0x7f0e780662e0 2024-03-24T11:18:40.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.892+0000 7f0e75d9b700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0e600692a0 0x7f0e6006b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-03-24T11:18:40.894 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.893+0000 7f0e75d9b700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0e600692a0 0x7f0e6006b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c004f40 tx=0x7f0e6c00b040 comp rx=0 tx=0).ready entity=mgr.14210 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-03-24T11:18:40.898 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:40.898+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mon.0 v2:172.21.15.100:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+176504 (secure 0 0 0) 0x7f0e68058240 con 0x7f0e780662e0 2024-03-24T11:18:41.124 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.123+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 --> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] -- mgr_command(tid 0: {"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}) v1 -- 0x7f0e7810f920 con 0x7f0e600692a0 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.125+0000 7f0e66ffd700 1 -- 172.21.15.100:0/2961944783 <== mgr.14210 v2:172.21.15.100:6800/2223271137 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+439 (secure 0 0 0) 0x7f0e7810f920 con 0x7f0e600692a0 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stdout:{ 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stdout: "target_image": "quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10", 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stdout: "in_progress": true, 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stdout: "which": "Upgrading daemons of type(s) mgr", 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stdout: "services_complete": [], 2024-03-24T11:18:41.125 INFO:teuthology.orchestra.run.smithi100.stdout: "progress": "0/2 daemons upgraded", 2024-03-24T11:18:41.126 INFO:teuthology.orchestra.run.smithi100.stdout: "message": "Pulling quay.ceph.io/ceph-ci/ceph@sha256:f3031fd9fa3ec502ba783e14433a4d3ebf8f66e5c31d12a3a8cd7e489d3d4d10 image on host smithi105", 2024-03-24T11:18:41.126 INFO:teuthology.orchestra.run.smithi100.stdout: "is_paused": false 2024-03-24T11:18:41.126 INFO:teuthology.orchestra.run.smithi100.stdout:} 2024-03-24T11:18:41.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0e600692a0 msgr2=0x7f0e6006b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:41.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0e600692a0 0x7f0e6006b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7f0e6c004f40 tx=0x7f0e6c00b040 comp rx=0 tx=0).stop 2024-03-24T11:18:41.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 msgr2=0x7f0e7810da30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-03-24T11:18:41.127 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e7810da30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7f0e68000c00 tx=0x7f0e68009e30 comp rx=0 tx=0).stop 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 shutdown_connections 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:6800/2223271137,v1:172.21.15.100:6801/2223271137] conn(0x7f0e600692a0 0x7f0e6006b750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.100:3300/0,v1:172.21.15.100:6789/0] conn(0x7f0e780662e0 0x7f0e7810da30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 --2- 172.21.15.100:0/2961944783 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0e780fff00 0x7f0e7810df70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 >> 172.21.15.100:0/2961944783 conn(0x7f0e780f7430 msgr2=0x7f0e780f7850 unknown :-1 s=STATE_NONE l=0).mark_down 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 shutdown_connections 2024-03-24T11:18:41.128 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:18:41.127+0000 7f0e7cccb700 1 -- 172.21.15.100:0/2961944783 wait complete. 2024-03-24T11:18:41.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:41 smithi100 ceph-mon[28941]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:41.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:41 smithi100 ceph-mon[28941]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:41.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:41 smithi100 ceph-mon[28941]: from='client.? 172.21.15.100:0/405013375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:18:41.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:41 smithi105 ceph-mon[36437]: from='client.24335 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:41.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:41 smithi105 ceph-mon[36437]: from='client.14588 -' entity='client.admin' cmd=[{"prefix": "orch ps", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:41.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:41 smithi105 ceph-mon[36437]: from='client.? 172.21.15.100:0/405013375' entity='client.admin' cmd=[{"prefix": "versions"}]: dispatch 2024-03-24T11:18:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:42 smithi105 ceph-mon[36437]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:42.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:42 smithi105 ceph-mon[36437]: pgmap v238: 65 pgs: 65 active+clean; 75 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-03-24T11:18:42.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:42 smithi100 ceph-mon[28941]: from='client.14596 -' entity='client.admin' cmd=[{"prefix": "orch upgrade status", "target": ["mon-mgr", ""]}]: dispatch 2024-03-24T11:18:42.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:42 smithi100 ceph-mon[28941]: pgmap v238: 65 pgs: 65 active+clean; 75 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 6.4 MiB/s wr, 507 op/s 2024-03-24T11:18:44.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: pgmap v239: 65 pgs: 65 active+clean; 83 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 512 op/s 2024-03-24T11:18:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: Upgrade: Updating mgr.smithi105.nmspmy 2024-03-24T11:18:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:18:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:18:44.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:44.554 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:44 smithi105 ceph-mon[36437]: Deploying daemon mgr.smithi105.nmspmy on smithi105 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: pgmap v239: 65 pgs: 65 active+clean; 83 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.7 MiB/s rd, 6.7 MiB/s wr, 512 op/s 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: Upgrade: Updating mgr.smithi105.nmspmy 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.nmspmy", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "mgr services"}]: dispatch 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-03-24T11:18:44.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:44 smithi100 ceph-mon[28941]: Deploying daemon mgr.smithi105.nmspmy on smithi105 2024-03-24T11:18:46.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:46 smithi105 ceph-mon[36437]: pgmap v240: 65 pgs: 65 active+clean; 92 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 554 op/s 2024-03-24T11:18:46.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:46 smithi100 ceph-mon[28941]: pgmap v240: 65 pgs: 65 active+clean; 92 MiB data, 2.0 GiB used, 534 GiB / 536 GiB avail; 2.2 MiB/s rd, 6.4 MiB/s wr, 554 op/s 2024-03-24T11:18:48.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:48 smithi105 ceph-mon[36437]: pgmap v241: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 511 op/s 2024-03-24T11:18:48.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:48 smithi100 ceph-mon[28941]: pgmap v241: 65 pgs: 65 active+clean; 102 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.6 MiB/s rd, 5.8 MiB/s wr, 511 op/s 2024-03-24T11:18:50.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:50 smithi105 ceph-mon[36437]: pgmap v242: 65 pgs: 65 active+clean; 121 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.9 MiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-03-24T11:18:50.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:50 smithi100 ceph-mon[28941]: pgmap v242: 65 pgs: 65 active+clean; 121 MiB data, 2.1 GiB used, 534 GiB / 536 GiB avail; 2.9 MiB/s rd, 5.3 MiB/s wr, 507 op/s 2024-03-24T11:18:52.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:52 smithi105 ceph-mon[36437]: pgmap v243: 65 pgs: 65 active+clean; 137 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 684 op/s 2024-03-24T11:18:52.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:52 smithi100 ceph-mon[28941]: pgmap v243: 65 pgs: 65 active+clean; 137 MiB data, 2.2 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 5.9 MiB/s wr, 684 op/s 2024-03-24T11:18:53.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:53 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:53.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:53 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-03-24T11:18:54.552 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:54 smithi105 ceph-mon[36437]: pgmap v244: 65 pgs: 65 active+clean; 145 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 767 op/s 2024-03-24T11:18:54.642 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:54 smithi100 ceph-mon[28941]: pgmap v244: 65 pgs: 65 active+clean; 145 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.5 MiB/s rd, 6.3 MiB/s wr, 767 op/s 2024-03-24T11:18:56.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:56 smithi105 ceph-mon[36437]: pgmap v245: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.2 MiB/s wr, 768 op/s 2024-03-24T11:18:56.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:56.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:56.553 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:56 smithi105 ceph-mon[36437]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:56.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:56 smithi100 ceph-mon[28941]: pgmap v245: 65 pgs: 65 active+clean; 153 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.3 MiB/s rd, 6.2 MiB/s wr, 768 op/s 2024-03-24T11:18:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:18:56.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:56 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-03-24T11:18:57.713 INFO:tasks.workunit.client.1.smithi105.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-24T11:18:57.742 INFO:tasks.workunit.client.1.smithi105.stdout:~/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.1/client.1/tmp/fsstress ~/cephtest/mnt.1/client.1/tmp 2024-03-24T11:18:57.743 INFO:tasks.workunit.client.1.smithi105.stderr:+ make 2024-03-24T11:18:58.804 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Mar 24 11:18:58 smithi105 ceph-mon[36437]: pgmap v246: 65 pgs: 65 active+clean; 157 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.8 MiB/s wr, 656 op/s 2024-03-24T11:18:58.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:58 smithi100 ceph-mon[28941]: pgmap v246: 65 pgs: 65 active+clean; 157 MiB data, 2.3 GiB used, 534 GiB / 536 GiB avail; 2.4 MiB/s rd, 5.8 MiB/s wr, 656 op/s 2024-03-24T11:18:59.626 INFO:tasks.workunit.client.1.smithi105.stdout:cc -DNO_XFS -I/home/ubuntu/cephtest/mnt.1/client.1/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress -D_LARGEFILE64_SOURCE -D_GNU_SOURCE -I../../../../include -I../../../../include -L../../../../lib fsstress.c -o fsstress 2024-03-24T11:19:00.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:59 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:19:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:59 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:19:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:59 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:19:00.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:18:59 smithi100 ceph-mon[28941]: from='mgr.14210 172.21.15.100:0/3941073038' entity='mgr.smithi100.smxgyi' 2024-03-24T11:19:00.658 INFO:tasks.workunit.client.0.smithi100.stderr:+ pushd ltp-full-20091231/testcases/kernel/fs/fsstress 2024-03-24T11:19:00.658 INFO:tasks.workunit.client.0.smithi100.stdout:~/cephtest/mnt.0/client.0/tmp/fsstress/ltp-full-20091231/testcases/kernel/fs/fsstress ~/cephtest/mnt.0/client.0/tmp/fsstress ~/cephtest/mnt.0/client.0/tmp 2024-03-24T11:19:00.658 INFO:tasks.workunit.client.0.smithi100.stderr:+ make 2024-03-24T11:19:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:23.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:23.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:23.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:23.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:23.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:24.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:24.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:24.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:25.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:25.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:25.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:25.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:25.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:25.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:25.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:25.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:25.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:26.236 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:25.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:26.236 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:25.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:26.236 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:25.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:26.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:26.236+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:26.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:26.236+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:26.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:26.236+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:26.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:26.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:26.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:26.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:27.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:27.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:27.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:26.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:26.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:26.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:27.828 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:27.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:27.828 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:27.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:27.829 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:27.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:27.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:27.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:27.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:28.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:28.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:28.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:28.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:28.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:28.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:28.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:28.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:28.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:28.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:29.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:29.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:29.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:28.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:28.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:28.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:29.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:29.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:29.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:30.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:30.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:30.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:30.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:30.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:30.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:30.554+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:30.554+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:30.554+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:31.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:31.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:31.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:31.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:31.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:31.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:31.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:31.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:31.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:32.278 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:32.024+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:32.278 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:32.024+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:32.278 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:32.024+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:32.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:32.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:32.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:32.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:32.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:32.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:32.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:32.582+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:32.851 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:32.582+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:32.851 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:32.582+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:33.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:33.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:33.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:33.562 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:33.265+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:33.562 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:33.265+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:33.562 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:33.265+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:33.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:33.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:33.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:34.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:33.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:34.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:33.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:34.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:33.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:34.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:34.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:34.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:34.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:34.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:34.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:34.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:34.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:35.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:35.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:35.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:35.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:35.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:35.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:35.597+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:35.597+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:35.597+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:36.246+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:36.246+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:36.246+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:36.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:36.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:36.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:36.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:36.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:36.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:37.237+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:37.237+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:37.237+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:37.087+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:37.087+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:37.087+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:37.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:37.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:37.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:37.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:38.269 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:38.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:38.269 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:38.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:38.269 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:38.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:38.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:38.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:38.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:38.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:38.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:38.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:38.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:38.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:38.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:39.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:39.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:39.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:39.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:39.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:39.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:39.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:39.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:39.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:40.040 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:39.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:40.040 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:39.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:40.040 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:39.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:40.240+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:40.240+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:40.240+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:40.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:40.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:40.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:40.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:40.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:40.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:40.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:40.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:40.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:41.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:40.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:41.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:40.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:41.257 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:40.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:41.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:41.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:41.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:41.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:42.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:41.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:42.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:41.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:42.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:41.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:42.303 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:42.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:42.303 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:42.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:42.303 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:42.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:42.302+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:42.302+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:42.302+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:42.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:42.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:42.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:42.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:42.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:42.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:43.287 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:43.045+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:43.287 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:43.045+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:43.287 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:43.045+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:43.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:43.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:43.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:43.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:43.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:43.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:44.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:43.700+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:44.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:43.700+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:44.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:43.700+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:44.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:44.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:44.307 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:44.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:44.307 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:44.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:44.306+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:44.306+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:44.306+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:45.062 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:44.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:45.062 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:44.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:45.062 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:44.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:45.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:45.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:45.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:45.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:45.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:45.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:46.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:45.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:46.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:45.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:46.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:45.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:46.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:46.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:46.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:46.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:46.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:46.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:47.061 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:46.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:47.062 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:46.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:47.062 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:46.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:47.322 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:47.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:47.322 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:47.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:47.322 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:47.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:47.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:47.322+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:47.322+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:47.322+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:48.087 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:47.731+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:48.087 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:47.731+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:48.087 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:47.731+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:48.086+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:48.086+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:48.086+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:48.321+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:48.321+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:48.321+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:49.080 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:48.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:49.080 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:48.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:49.080 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:48.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:49.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:49.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:49.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:49.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:49.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:49.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:50.090 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:49.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:50.090 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:49.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:50.090 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:49.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:50.089+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:50.089+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:50.089+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:50.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:50.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:50.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:50.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:50.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:50.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:51.054+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:51.054+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:51.054+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:51.321+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:51.321+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:51.321+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:51.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:51.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:51.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:52.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:52.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:52.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:52.362+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:52.362+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:52.362+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:52.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:52.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:52.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:53.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:53.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:53.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:53.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:53.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:53.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:53.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:53.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:53.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:53.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:53.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:53.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:54.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:54.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:54.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:54.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:54.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:54.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:54.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:54.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:54.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:54.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:54.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:54.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:55.815 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:55.408+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:55.815 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:55.408+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:55.815 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:55.408+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:55.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:55.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:55.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:55.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:55.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:55.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:56.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:56.436+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:56.767 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:56.436+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:56.767 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:56.436+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:56.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:56.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:56.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:56.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:56.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:56.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:57.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:57.399+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:57.778 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:57.399+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:57.778 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:57.399+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:57.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:57.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:57.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:57.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:57.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:57.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:58.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:58.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:58.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:58.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:58.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:58.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:19:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:58.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:58.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:58.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:19:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:59.377+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:59.378+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:19:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:19:59.378+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:59.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:59.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:19:59.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:59.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:59.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:19:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:19:59.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:00.742 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:00.412+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:00.742 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:00.412+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:00.743 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:00.412+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:00.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:00.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:00.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:00.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:00.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:00.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:01.719 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:01.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:01.719 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:01.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:01.719 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:01.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:01.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:01.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:01.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:01.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:01.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:01.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:02.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:02.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:02.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:02.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:02.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:02.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:02.749+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:02.749+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:02.749+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:02.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:02.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:02.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:03.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:03.501+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:03.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:03.501+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:03.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:03.501+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:03.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:03.748+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:03.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:03.748+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:03.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:03.748+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:04.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:03.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:03.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:03.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:04.546+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:04.546+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:04.546+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:04.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:04.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:04.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:04.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:04.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:04.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:04.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:04.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:05.713+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:05.713+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:05.713+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:05.590+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:05.590+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:05.590+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:05.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:05.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:05.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:05.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:06.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:06.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:06.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:06.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:06.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:06.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:06.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:06.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:06.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:07.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:07.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:07.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:07.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:07.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:07.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:07.865 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:07.570+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:07.865 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:07.570+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:07.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:07.570+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:07.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:07.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:07.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:07.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:07.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:07.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:08.794 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:08.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:08.794 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:08.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:08.794 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:08.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:08.794 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:08.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:08.794 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:08.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:08.795 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:08.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:09.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:08.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:08.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:08.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:09.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:09.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:09.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:09.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:09.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:09.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:09.584+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:09.584+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:09.584+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:10.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:10.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:10.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:10.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:10.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:10.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:10.557+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:10.557+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:10.557+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:11.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:11.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:11.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:11.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:11.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:11.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:11.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:11.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:11.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:12.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:12.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:12.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:12.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:12.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:12.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:12.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:12.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:12.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:12.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:12.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:12.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:12.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:12.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:12.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:13.833 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:13.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:13.833 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:13.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:13.833 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:13.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:13.833 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:13.758+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:13.834 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:13.758+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:13.834 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:13.758+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:14.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:13.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:13.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:13.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:14.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:14.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:14.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:14.642+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:14.642+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:14.642+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:14.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:14.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:14.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:15.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:15.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:15.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:15.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:15.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:15.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:15.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:15.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:15.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:16.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:16.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:16.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:16.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:16.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:16.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:16.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:16.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:16.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:17.641+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:17.641+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:17.641+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:17.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:17.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:17.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:17.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:17.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:17.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:18.612+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:18.612+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:18.612+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:18.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:18.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:18.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:18.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:18.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:18.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:18.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:18.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:18.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:19.574+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:19.574+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:19.574+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:19.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:19.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:19.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:19.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:19.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:19.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:19.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:20.809 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:20.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:20.809 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:20.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:20.809 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:20.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:20.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:20.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:20.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:20.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:20.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:20.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:21.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:20.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:20.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:20.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:21.813 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:21.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:21.813 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:21.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:21.813 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:21.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:21.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:21.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:21.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:21.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:21.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:21.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:22.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:22.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:22.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:22.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:22.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:22.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:22.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:22.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:22.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:22.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:22.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:22.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:22.867 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:22.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:22.867 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:22.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:22.867 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:22.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:23.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:23.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:23.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:23.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:23.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:23.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:23.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:23.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:23.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:23.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:23.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:23.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:24.834 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:24.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:24.834 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:24.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:24.834 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:24.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:24.834 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:24.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:24.834 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:24.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:24.834 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:24.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:25.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:24.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:24.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:24.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:25.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:25.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:25.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:25.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:25.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:25.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:25.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:25.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:25.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:25.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:26.861 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:26.564+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:26.861 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:26.564+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:26.861 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:26.564+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:26.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:26.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:26.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:26.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:26.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:26.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:27.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:27.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:27.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:27.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:27.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:27.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:27.909+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:27.909+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:27.909+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:27.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:27.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:27.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:28.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:28.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:28.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:28.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:28.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:28.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:28.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:28.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:28.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:28.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:28.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:28.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:29.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:29.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:29.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:29.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:29.848 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:29.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:29.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:29.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:29.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:29.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:29.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:29.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:30.821 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:30.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:30.821 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:30.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:30.821 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:30.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:30.820+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:30.820+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:30.820+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:30.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:30.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:30.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:31.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:31.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:31.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:31.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:31.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:31.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:31.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:31.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:31.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:32.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:32.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:32.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:32.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:32.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:32.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:32.981 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:32.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:32.981 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:32.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:32.981 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:32.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:32.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:32.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:32.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:32.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:32.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:32.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:33.838 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:33.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:33.838 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:33.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:33.838 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:33.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:33.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:33.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:33.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:33.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:33.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:33.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:34.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:34.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:34.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:34.818+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:34.818+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:34.818+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:34.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:34.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:34.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:35.948 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:35.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:35.948 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:35.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:35.948 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:35.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:35.948 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:35.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:35.949 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:35.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:35.949 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:35.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:35.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:35.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:35.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:36.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:36.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:36.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:36.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:36.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:36.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:36.971+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:36.971+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:36.971+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:37.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:37.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:37.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:37.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:37.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:37.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:37.980 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:37.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:37.980 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:37.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:37.980 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:37.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:38.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:38.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:38.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:38.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:38.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:38.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:38.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:38.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:38.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:39.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:39.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:39.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:39.988 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:39.792+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:39.989 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:39.792+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:39.989 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:39.792+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:39.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:39.713+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:39.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:39.713+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:39.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:39.713+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:40.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:39.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:39.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:39.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:40.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:40.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:40.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:40.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:40.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:40.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:40.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:40.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:40.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:41.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:41.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:41.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:41.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:41.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:41.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:41.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:41.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:41.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:41.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:42.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:42.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:42.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:42.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:42.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:42.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:42.984 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:42.711+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:42.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:42.711+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:42.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:42.711+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:42.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:42.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:42.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:42.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:42.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:42.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:43.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:43.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:43.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:43.665+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:43.665+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:43.665+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:43.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:43.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:43.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:44.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:44.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:44.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:44.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:44.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:44.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:44.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:44.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:44.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:45.963 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:45.692+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:45.963 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:45.692+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:45.963 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:45.692+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:45.964 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:45.659+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:45.964 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:45.659+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:45.964 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:45.659+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:45.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:45.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:45.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:46.992 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:46.662+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:46.992 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:46.662+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:46.993 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:46.662+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:46.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:46.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:46.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:46.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:46.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:46.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:46.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:46.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:46.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:47.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:47.737+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:47.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:47.737+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:47.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:47.737+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:47.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:47.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:47.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:47.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:47.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:47.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:48.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:48.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:48.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:48.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:48.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:48.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:48.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:48.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:48.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:48.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:48.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:48.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:48.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:48.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:49.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:49.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:49.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:49.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:49.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:49.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:49.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:49.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:49.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:50.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:50.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:50.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:50.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:50.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:50.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:50.909+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:50.909+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:50.909+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:51.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:51.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:51.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:51.635+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:51.635+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:51.635+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:51.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:51.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:51.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:52.861 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:52.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:52.862 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:52.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:52.862 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:52.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:52.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:52.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:52.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:52.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:52.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:52.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:52.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:52.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:52.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:53.820 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:53.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:53.820 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:53.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:53.820 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:53.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:53.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:53.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:53.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:53.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:53.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:53.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:55.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:54.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:55.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:54.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:55.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:54.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:55.008 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:54.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:55.008 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:54.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:55.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:54.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:55.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:54.797+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:55.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:54.797+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:55.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:54.797+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:55.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:55.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:55.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:55.813+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:55.813+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:56.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:55.813+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:55.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:55.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:55.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:56.643+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:56.643+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:56.643+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:56.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:56.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:56.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:56.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:56.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:56.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:57.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:57.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:57.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:57.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:57.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:57.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:57.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:57.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:57.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:57.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:57.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:57.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:57.873 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:57.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:57.873 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:57.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:57.873 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:57.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:58.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:58.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:58.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:58.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:58.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:58.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:58.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:58.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:58.619+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:59.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:59.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:20:59.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:59.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:59.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:20:59.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:20:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:59.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:59.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:20:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:20:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:20:59.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:00.860 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:00.585+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:00.860 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:00.585+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:00.860 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:00.585+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:00.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:00.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:00.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:00.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:00.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:00.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:01.835 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:01.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:01.835 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:01.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:01.835 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:01.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:01.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:01.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:01.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:01.952+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:01.952+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:01.952+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:02.806 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:02.543+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:02.807 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:02.543+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:02.807 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:02.543+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:03.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:02.859+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:02.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:02.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:02.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:02.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:02.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:02.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:03.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:03.508+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:03.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:03.508+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:03.763 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:03.508+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:03.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:03.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:03.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:03.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:03.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:03.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:04.466+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:04.466+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:04.466+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:04.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:04.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:04.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:04.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:04.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:04.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:05.738 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:05.465+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:05.738 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:05.465+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:05.738 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:05.465+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:05.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:05.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:05.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:05.737+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:05.737+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:05.737+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:06.756 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:06.452+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:06.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:06.452+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:06.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:06.452+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:06.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:06.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:06.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:06.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:06.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:06.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:07.786 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:07.411+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:07.786 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:07.411+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:07.786 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:07.411+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:08.144 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:07.860+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:08.206 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:07.845+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:08.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:07.845+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:08.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:07.845+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:08.207 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:07.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:08.207 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:07.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:08.207 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:07.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:08.803 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:08.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:08.803 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:08.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:08.803 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:08.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:08.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:08.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:08.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:08.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:08.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:08.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:09.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:09.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:09.799 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:09.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:09.799 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:09.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:09.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:09.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:09.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:09.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:09.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:09.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:10.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:10.470+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:10.774 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:10.470+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:10.774 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:10.470+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:10.803+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:10.803+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:10.803+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:10.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:10.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:10.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:11.806 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:11.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:11.806 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:11.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:11.806 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:11.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:11.806 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:11.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:11.807 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:11.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:11.807 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:11.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:11.805+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:11.805+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:11.805+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:12.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:12.443+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:12.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:12.443+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:12.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:12.443+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:13.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:12.861+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:12.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:12.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:12.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:12.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:12.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:12.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:13.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:13.454+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:13.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:13.454+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:13.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:13.454+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:13.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:13.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:13.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:13.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:13.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:13.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:14.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:14.417+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:14.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:14.417+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:14.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:14.417+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:14.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:14.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:14.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:14.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:14.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:14.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:15.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:15.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:15.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:15.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:15.777 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:15.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:15.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:15.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:15.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:15.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:15.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:15.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:16.740 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:16.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:16.741 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:16.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:16.741 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:16.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:16.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:16.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:16.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:16.740+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:16.740+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:16.740+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:17.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:17.479+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:17.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:17.479+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:17.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:17.479+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:18.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:17.861+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:17.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:17.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:17.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:17.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:17.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:18.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:17.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:18.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:18.519+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:18.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:18.519+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:18.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:18.519+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:18.824 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:18.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:18.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:18.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:18.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:18.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:18.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:18.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:18.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:19.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:19.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:19.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:19.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:19.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:19.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:19.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:19.532+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:19.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:19.532+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:19.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:19.532+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:20.743+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:20.743+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:20.743+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:20.738+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:20.738+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:20.738+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:20.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:20.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:20.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:20.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:20.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:20.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:21.761 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:21.495+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:21.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:21.495+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:21.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:21.495+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:21.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:21.710+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:21.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:21.710+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:21.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:21.710+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:21.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:21.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:21.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:22.863 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:22.545+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:22.863 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:22.545+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:22.863 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:22.545+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:22.863 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:22.718+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:22.863 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:22.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:22.863 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:22.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:22.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:22.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:22.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:23.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:22.862+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 51 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:23.705+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:23.705+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:23.705+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:23.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:23.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:23.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:23.677+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:23.677+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:23.677+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:24.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:24.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:24.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:24.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:24.617+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:24.617+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:24.617+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:24.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:24.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:24.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:25.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:25.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:25.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:25.595+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:25.595+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:25.595+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:25.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:25.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:25.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:26.726+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:26.726+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:26.726+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:26.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:26.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:26.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:26.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:26.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:26.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:26.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:26.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:26.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:27.987 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:27.863+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 83 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:27.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:27.661+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:27.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:27.661+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:27.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:27.661+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:27.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:27.733+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:27.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:27.733+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:27.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:27.733+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:27.988 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:27.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:27.988 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:27.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:27.988 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:27.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:28.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:28.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:28.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:28.652+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:28.652+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:28.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:28.652+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:28.694+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:28.694+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:28.694+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:29.648+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:29.648+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:29.648+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:29.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:29.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:29.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:29.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:29.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:29.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:29.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:29.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:30.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:30.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:30.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:30.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:30.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:30.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:30.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:30.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:30.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:30.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:31.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:31.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:31.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:31.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:31.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:31.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:31.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:31.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:31.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:32.876 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:32.863+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 112 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:32.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:32.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:32.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:32.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:32.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:32.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:32.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:32.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:32.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:32.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:32.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:32.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:32.878 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:32.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:32.878 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:32.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:32.878 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:32.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:33.792 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:33.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:33.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:33.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:33.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:33.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:33.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:33.590+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:33.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:33.590+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:33.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:33.590+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:33.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:33.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:33.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:34.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:34.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:34.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:34.684+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:34.684+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:34.684+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:34.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:34.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:34.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:35.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:35.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:35.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:35.711+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:35.711+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:35.711+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:35.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:35.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:35.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:36.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:36.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:36.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:36.672+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:36.672+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:36.672+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:36.814+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:36.814+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:36.814+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:37.989 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:37.864+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 142 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:37.989 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:37.720+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:37.989 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:37.720+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:37.989 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:37.720+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:37.990 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:37.696+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:37.990 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:37.696+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:37.990 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:37.696+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:37.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:37.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:37.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:37.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:37.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:37.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:38.737+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:38.737+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:38.737+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:38.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:38.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:38.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:39.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:38.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:39.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:38.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:39.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:38.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:39.719+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:39.719+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:39.719+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:39.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:39.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:39.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:39.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:39.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:39.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:40.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:40.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:40.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:40.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:40.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:40.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:40.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:40.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:40.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:41.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:41.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:41.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:41.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:41.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:41.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:41.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:41.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:41.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:41.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:41.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:41.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:43.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:42.863+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 173 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:42.793+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:42.793+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:42.793+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:42.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:42.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:42.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:42.755+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:42.755+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:42.755+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:43.789 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:43.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:43.789 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:43.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:43.789 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:43.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:43.828+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:43.828+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:43.828+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:43.788+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:43.788+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:43.788+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:44.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:44.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:44.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:44.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:44.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:44.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:44.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:44.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:44.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:45.821+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:45.821+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:45.821+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:45.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:45.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:45.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:45.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:45.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:45.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:46.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:46.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:46.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:46.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:46.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:46.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:46.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:46.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:46.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:47.990 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:47.865+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 204 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:47.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:47.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:47.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:47.678+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:47.678+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:47.678+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:47.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:47.991 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:47.649+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:47.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:47.649+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:48.620+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:48.620+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:48.620+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:48.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:48.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:48.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:48.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:48.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:48.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:48.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:48.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:48.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:49.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:49.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:49.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:49.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:49.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:49.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:49.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:49.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:49.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:49.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:49.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:50.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:50.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:50.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:50.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:50.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:50.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:50.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:50.565+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:50.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:50.565+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:50.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:50.565+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:50.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:50.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:50.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:51.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:51.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:51.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:51.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:51.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:51.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:51.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:51.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:51.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:51.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:51.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:51.626+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:52.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:52.589+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:52.589+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:52.589+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:52.647+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:52.647+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:52.647+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:52.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:52.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:52.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:52.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:53.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:52.865+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 232 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:53.607+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:53.607+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:53.607+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:53.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:53.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:53.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:53.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:53.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:53.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:54.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:54.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:54.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:54.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:54.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:54.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:54.647+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:54.647+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:54.647+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:55.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:55.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:55.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:55.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:55.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:55.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:55.824 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:55.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:55.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:55.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:55.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:55.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:55.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:55.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:55.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:56.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:56.644+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:56.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:56.644+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:56.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:56.644+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:56.847 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:56.567+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:56.847 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:56.567+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:56.847 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:56.567+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:57.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:56.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:56.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:56.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:57.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:57.657+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:57.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:57.657+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:57.867 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:57.657+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:57.868 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:57.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:57.868 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:57.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:57.868 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:57.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:58.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:21:57.866+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 261 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:21:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:57.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:57.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:57.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:58.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:58.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:58.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:58.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:58.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:58.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:58.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:58.614+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:58.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:58.614+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:58.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:58.614+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:59.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:58.873+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:58.873+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:58.873+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:59.658+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:59.658+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:21:59.658+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:21:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:59.593+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:59.593+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:21:59.593+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:21:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:59.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:59.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:21:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:21:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:21:59.828+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:00.865 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:00.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:00.865 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:00.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:00.865 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:00.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:00.865 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:00.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:00.865 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:00.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:00.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:00.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:00.864+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:00.864+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:00.864+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:01.849 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:01.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:01.849 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:01.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:01.849 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:01.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:01.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:01.581+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:01.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:01.581+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:01.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:01.581+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:02.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:01.848+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:01.848+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:01.848+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:02.868 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:02.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:02.868 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:02.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:02.868 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:02.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:02.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:02.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:02.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:02.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:02.868 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:02.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:03.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:02.867+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 293 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:02.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:02.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:02.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:03.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:03.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:03.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:03.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:03.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:03.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:03.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:03.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:03.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:03.894+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:03.894+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:03.894+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:04.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:04.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:04.882 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:04.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:04.882 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:04.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:04.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:04.571+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:04.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:04.571+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:04.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:04.571+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:04.881+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:04.881+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:04.881+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:05.878 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:05.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:05.878 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:05.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:05.878 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:05.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:05.878 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:05.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:05.879 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:05.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:05.879 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:05.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:06.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:05.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:05.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:05.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:06.517+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:06.517+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:06.517+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:06.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:06.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:06.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:06.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:06.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:06.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:07.862 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:07.502+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:07.862 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:07.502+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:07.862 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:07.502+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:07.862 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:07.555+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:07.862 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:07.555+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:07.863 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:07.555+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:08.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:07.868+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 323 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:08.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:07.862+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:08.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:07.862+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:08.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:07.862+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:08.510+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:08.510+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:08.510+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:08.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:08.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:08.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:08.903+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:08.903+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:08.903+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:09.570+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:09.570+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:09.570+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:09.524+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:09.524+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:09.524+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:09.916+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:09.916+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:09.916+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:10.869 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:10.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:10.870 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:10.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:10.870 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:10.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:10.870 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:10.536+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:10.870 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:10.536+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:10.870 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:10.536+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:11.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:10.869+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:10.869+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:10.869+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:11.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:11.556+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:11.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:11.556+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:11.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:11.556+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:11.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:11.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:11.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:11.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:11.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:11.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:11.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:11.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:11.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:12.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:12.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:12.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:12.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:12.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:12.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:12.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:12.515+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:12.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:12.515+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:12.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:12.515+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:13.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:12.868+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 351 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:12.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:12.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:12.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:13.832 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:13.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:13.832 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:13.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:13.832 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:13.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:13.832 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:13.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:13.832 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:13.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:13.832 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:13.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:13.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:13.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:13.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:14.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:14.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:14.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:14.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:14.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:14.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:14.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:14.436+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:14.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:14.436+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:14.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:14.436+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:14.871+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:14.871+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:14.871+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:15.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:15.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:15.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:15.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:15.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:15.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:15.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:15.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:15.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:15.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:15.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:15.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:15.876+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:15.876+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:15.876+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:16.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:16.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:16.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:16.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:16.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:16.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:16.902+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:16.902+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:16.902+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:17.870 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:17.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:17.870 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:17.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:17.870 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:17.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:17.870 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:17.467+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:17.870 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:17.467+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:17.870 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:17.467+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:18.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:17.869+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 380 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:17.910+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:17.910+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:17.910+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:18.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:18.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:18.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:18.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:18.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:18.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:18.919+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:18.919+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:18.919+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:19.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:19.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:19.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:19.500+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:19.500+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:19.500+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:19.924+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:19.924+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:19.924+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:20.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:20.479+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:20.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:20.479+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:20.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:20.479+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:20.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:20.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:20.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:20.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:20.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:20.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:20.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:20.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:20.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:21.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:21.435+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:21.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:21.435+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:21.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:21.435+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:21.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:21.499+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:21.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:21.499+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:21.866 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:21.499+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:21.866+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:21.866+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:21.866+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:22.827 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:22.433+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:22.828 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:22.433+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:22.828 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:22.433+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:22.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:22.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:22.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:22.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:22.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:22.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:23.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:22.870+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 410 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:22.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:22.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:22.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:23.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:23.472+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:23.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:23.472+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:23.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:23.472+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:23.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:23.498+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:23.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:23.498+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:23.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:23.498+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:24.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:23.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:23.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:23.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:24.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:24.455+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:24.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:24.455+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:24.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:24.455+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:24.885 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:24.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:24.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:24.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:24.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:24.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:25.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:24.884+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:24.884+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:24.884+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:25.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:25.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:25.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:25.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:25.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:25.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:25.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:25.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:25.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:26.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:26.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:26.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:26.450+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:26.450+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:26.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:26.450+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:26.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:26.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:26.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:27.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:27.421+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:27.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:27.421+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:27.871 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:27.421+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:27.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:27.432+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:27.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:27.432+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:27.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:27.432+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:28.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:27.869+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 439 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:27.935+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:27.935+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:27.935+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:28.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:28.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:28.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:28.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:28.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:28.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:28.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:28.388+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:28.460+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:28.460+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:28.460+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:28.461+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:28.900+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:28.900+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:28.900+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:28.900+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:29.855 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:29.405+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:29.855 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:29.405+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:29.855 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:29.405+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:29.855 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:29.405+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:29.855 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:29.473+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:29.855 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:29.473+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:29.856 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:29.473+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:29.856 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:29.473+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:29.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:29.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:29.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:29.855+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:30.399+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:30.399+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:30.399+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:30.399+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:30.426+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:30.426+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:30.426+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:30.426+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:30.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:30.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:30.846+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:30.847+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:31.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:31.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:31.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:31.446+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:31.362+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:31.362+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:31.362+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:31.635 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:31.362+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:32.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:31.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:32.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:31.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:32.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:31.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:32.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:31.893+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:32.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:32.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:32.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:32.368+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:32.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:32.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:32.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:32.469+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:33.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:32.870+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 467 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:32.938+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:32.938+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:32.938+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:32.938+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:33.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:33.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:33.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:33.327+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:33.422+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:33.422+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:33.422+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:33.423+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:34.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:33.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:34.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:33.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:34.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:33.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:34.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:33.894+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:34.421+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:34.421+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:34.421+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:34.422+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:34.328+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:34.328+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:34.328+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:34.328+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:34.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:34.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:34.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:34.861+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:35.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:35.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:35.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:35.469+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:35.291+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:35.291+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:35.291+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:35.291+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:35.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:35.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:35.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:35.846+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:36.476+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:36.476+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:36.476+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:36.476+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:36.336+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:36.336+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:36.336+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:36.336+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:36.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:36.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:36.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:36.813+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:37.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:37.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:37.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:37.468+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:37.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:37.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:37.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:37.375+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:38.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:37.871+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 496 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:37.818+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:37.818+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:37.818+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:37.818+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:38.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:38.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:38.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:38.428+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:38.329+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:38.329+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:38.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:38.329+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:38.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:38.329+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:38.867+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:38.867+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:38.867+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:38.868+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:39.458+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:39.458+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:39.458+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:39.458+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:39.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:39.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:39.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:39.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:39.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:39.331+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:40.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:39.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:40.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:39.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:40.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:39.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:40.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:39.905+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:40.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:40.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:40.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:40.478+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:40.307+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:40.307+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:40.307+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:40.308+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:40.886+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:40.886+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:40.886+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:40.887+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:41.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:41.520+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:41.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:41.520+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:41.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:41.520+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:41.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:41.520+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:41.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:41.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:41.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:41.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:41.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:41.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:41.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:41.312+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:41.850+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:41.850+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:41.850+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:41.850+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:42.553 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:42.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:42.553 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:42.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:42.553 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:42.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:42.553 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:42.293+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:42.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:42.552+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:42.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:42.552+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:42.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:42.552+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:42.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:42.552+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:43.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:42.872+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 526 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:42.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:42.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:42.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:42.878+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:43.525 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:43.507+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:43.525 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:43.507+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:43.525 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:43.507+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:43.525 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:43.508+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:43.525 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:43.266+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:43.526 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:43.266+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:43.526 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:43.266+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:43.526 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:43.266+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:44.219 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:43.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:44.220 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:43.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:44.220 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:43.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:44.220 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:43.908+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:44.536 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:44.219+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:44.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:44.219+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:44.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:44.219+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:44.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:44.219+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:44.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:44.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:44.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:44.536+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:45.212 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:44.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:45.213 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:44.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:45.213 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:44.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:45.213 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:44.927+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:45.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:45.212+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:45.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:45.212+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:45.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:45.212+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:45.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:45.212+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:45.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:45.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:45.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:45.527+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:45.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:45.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:45.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:45.971+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:46.187+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:46.187+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:46.187+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:46.187+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:46.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:46.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:46.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:46.536+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:47.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:46.962+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:47.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:46.962+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:47.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:46.962+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:47.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:46.963+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:47.493 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:47.224+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:47.494 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:47.224+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:47.494 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:47.224+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:47.494 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:47.224+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:47.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:47.493+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:47.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:47.493+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:47.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:47.493+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:47.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:47.493+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:47.872+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 557 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:47.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:47.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:47.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:47.928+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:48.473 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:48.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:48.473 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:48.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:48.473 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:48.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:48.473 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:48.199+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:48.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:48.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:48.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:48.472+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:49.227 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:48.931+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:49.228 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:48.931+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:49.228 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:48.931+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:49.228 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:48.932+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:49.511 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:49.227+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:49.511 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:49.227+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:49.511 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:49.227+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:49.511 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:49.227+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:49.510+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:49.510+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:49.510+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:49.510+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:50.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:49.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:50.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:49.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:50.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:49.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:50.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:49.982+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:50.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:50.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:50.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:50.265+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:50.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:50.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:50.468+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:50.469+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:51.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:50.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:51.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:50.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:51.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:50.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:51.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:50.985+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:51.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:51.308+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:51.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:51.308+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:51.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:51.308+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:51.636 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:51.308+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:51.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:51.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:51.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:51.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:51.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:51.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:51.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:51.493+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:52.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:51.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:52.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:51.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:52.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:51.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:52.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:51.948+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:52.495+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:52.495+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:52.495+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:52.495+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:52.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:52.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:52.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:52.330+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:53.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:52.873+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 585 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:52.946+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:52.946+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:52.946+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:52.946+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:53.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:53.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:53.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:53.459+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:53.356+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:53.356+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:53.356+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:53.356+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:54.322 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:53.956+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:54.322 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:53.956+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:54.322 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:53.956+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:54.323 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:53.956+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:54.477+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:54.477+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:54.477+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:54.477+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:54.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:54.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:54.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:54.322+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:55.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:55.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:55.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:55.005+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:55.451+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:55.451+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:55.451+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:55.451+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:55.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:55.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:55.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:55.346+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:56.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:56.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:56.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:56.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:56.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:56.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:56.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:56.008+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:56.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:56.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:56.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:56.485+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:56.382+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:56.382+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:56.382+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:56.382+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:57.341 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:57.341 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:57.341 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:57.341 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.007+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:57.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:57.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:57.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:57.488+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:57.341+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:57.341+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:57.341+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:57.341+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:58.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:22:57.873+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 629 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:22:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.998+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.998+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.998+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:57.999+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:58.508 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:58.356+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:58.508 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:58.356+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:58.509 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:58.356+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:58.509 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:58.356+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:58.508+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:58.508+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:58.508+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:58.508+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:59.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:59.012+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:59.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:59.012+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:59.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:59.012+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:22:59.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:22:59.012+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:22:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:59.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:59.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:59.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:22:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:22:59.515+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:22:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:59.316+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:59.316+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:59.316+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:22:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:22:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:22:59.316+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:00.298 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:00.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:00.298 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:00.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:00.298 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:00.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:00.298 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:00.020+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:00.561 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:00.298+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:00.561 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:00.298+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:00.561 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:00.298+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:00.561 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:00.298+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:00.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:00.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:00.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:00.560+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:01.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:01.053+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:01.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:01.053+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:01.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:01.053+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:01.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:01.053+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:01.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:01.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:01.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:01.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:01.536+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:01.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:01.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:01.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:01.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:01.320+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:02.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:02.052+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:02.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:02.052+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:02.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:02.052+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:02.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:02.053+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:02.558+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:02.558+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:02.558+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:02.558+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:02.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:02.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:02.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:02.320+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:03.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:02.874+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 659 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:03.082+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:03.082+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:03.082+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:03.082+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:03.547 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:03.289+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:03.548 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:03.289+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:03.548 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:03.289+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:03.548 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:03.289+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:03.547+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:03.547+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:03.547+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:03.547+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:04.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:04.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:04.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:04.263+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:04.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:04.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:04.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:04.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:04.064+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:04.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:04.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:04.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:04.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:04.529+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:05.256+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:05.256+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:05.256+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:05.256+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:05.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:05.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:05.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:05.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:05.072+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:05.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:05.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:05.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:05.536+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:06.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:06.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:06.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:06.098+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:06.235+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:06.235+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:06.235+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:06.235+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:06.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:06.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:06.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:06.567+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:07.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:07.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:07.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:07.096+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:07.250+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:07.250+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:07.250+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:07.251+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:07.875 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:07.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:07.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:07.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:07.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:07.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:07.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:07.615+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:08.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:07.874+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 688 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:08.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:08.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:08.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:08.144+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:08.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:08.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:08.581 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:08.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:08.582 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:08.263+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:08.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:08.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:08.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:08.580+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:09.249 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:09.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:09.249 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:09.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:09.249 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:09.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:09.249 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:09.154+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:09.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:09.248+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:09.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:09.248+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:09.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:09.248+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:09.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:09.248+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:09.537+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:09.537+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:09.537+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:09.537+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:10.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:10.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:10.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:10.208+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:10.135+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:10.135+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:10.135+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:10.135+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:10.547+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:10.547+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:10.547+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:10.548+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:11.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:11.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:11.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:11.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:11.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:11.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:11.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:11.145+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:11.550 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:11.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:11.550 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:11.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:11.551 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:11.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:11.551 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:11.210+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:11.548+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:11.548+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:11.548+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:11.548+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:12.112+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:12.112+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:12.112+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:12.112+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:12.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:12.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:12.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:12.211+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:12.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:12.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:12.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:12.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:12.523+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:13.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:12.875+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 716 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:13.105+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:13.105+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:13.105+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:13.105+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:13.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:13.249+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:13.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:13.249+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:13.527 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:13.249+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:13.528 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:13.249+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:13.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:13.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:13.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:13.564+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:14.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:14.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:14.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:14.092+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:14.215+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:14.215+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:14.215+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:14.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:14.215+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:14.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:14.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:14.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:14.589+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:15.188+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:15.188+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:15.188+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:15.188+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:15.080+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:15.080+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:15.080+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:15.081+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:15.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:15.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:15.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:15.606+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:16.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:16.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:16.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:16.199+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:16.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:16.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:16.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:16.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:16.056+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:16.572+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:16.572+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:16.572+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:16.572+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:17.178+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:17.178+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:17.178+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:17.179+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:17.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:17.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:17.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:17.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:17.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:17.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:17.056+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:17.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:17.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:17.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:17.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:17.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:17.580+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:17.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:17.581+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:18.139 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:17.876+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 745 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:18.139 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:18.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:18.139 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:18.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:18.139 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:18.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:18.139 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:18.094+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:18.138+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:18.138+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:18.138+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:18.138+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:18.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:18.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:18.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:18.624+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:19.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:19.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:19.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:19.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:19.106+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:19.080+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:19.080+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:19.080+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:19.081+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:20.033 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:19.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:20.034 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:19.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:20.034 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:19.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:20.034 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:19.675+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.033+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:20.130+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:20.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:20.130+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:20.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:20.130+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:20.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:20.130+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:20.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:20.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:20.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:20.641+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.992+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.992+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.992+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:20.992+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:21.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:21.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:21.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:21.090+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:21.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:21.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:21.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:21.675+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:21.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:21.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:21.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:21.988+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:22.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:22.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:22.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:22.080+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:22.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:22.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:22.878 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:22.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:22.878 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:22.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:22.878 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:22.704+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:23.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:22.876+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 775 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:22.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:22.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:22.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:22.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:23.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:23.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:23.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:23.106+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:23.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:23.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:23.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:23.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:23.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:23.673+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:23.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:23.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:23.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:23.964+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:24.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:24.127+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:24.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:24.127+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:24.127+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:24.127+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:25.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:24.712+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:25.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:24.712+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:25.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:24.712+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:25.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:24.712+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:25.128 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:24.960+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:25.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:24.960+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:25.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:24.960+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:25.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:24.960+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:25.127+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:25.127+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:25.127+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:25.127+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:25.747+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:25.747+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:25.747+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:25.747+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:26.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:26.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:26.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:26.167+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:25.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:25.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:26.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:25.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:26.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:25.983+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:26.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:26.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:26.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:26.717+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:26.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:27.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:26.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:27.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:26.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:27.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:26.939+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:27.178+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:27.178+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:27.178+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:27.178+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:27.968 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:27.877+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 804 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:27.968 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:27.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:27.968 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:27.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:27.968 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:27.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:27.968 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:27.672+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:28.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:27.967+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:28.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:27.967+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:28.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:27.967+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:28.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:27.968+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:28.157+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:28.157+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:28.157+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:28.157+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:28.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:28.651+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:28.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:28.651+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:28.651+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:28.651+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:28.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:28.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:28.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:28.985+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:29.123+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:29.123+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:29.123+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:29.123+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:29.636+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:29.636+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:29.636+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:29.636+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:30.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:30.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:30.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:30.031+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:30.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:30.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:30.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:30.083+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:30.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:30.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:30.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:30.621+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:31.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:31.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:31.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:31.039+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:31.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:31.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:31.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:31.110+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:31.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:31.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:31.581+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:31.581+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:32.042+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:32.042+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:32.042+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:32.042+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:32.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:32.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:32.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:32.080+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:32.879 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:32.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:32.879 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:32.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:32.879 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:32.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:32.879 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:32.623+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:33.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:32.878+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 832 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:33.012+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:33.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:33.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:33.013+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:33.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:33.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:33.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:33.081+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:33.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:33.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:33.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:33.672+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:34.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:34.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:34.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:34.040+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:34.097+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:34.097+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:34.097+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:34.098+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:35.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:34.653+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:35.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:34.653+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:35.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:34.653+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:35.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:34.653+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:35.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:35.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:35.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:35.059+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:35.130+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:35.130+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:35.130+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:35.130+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:36.063 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:35.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:36.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:35.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:36.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:35.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:36.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:35.695+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:36.105+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:36.105+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:36.105+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:36.106+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:36.062+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:36.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:36.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:36.063+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:37.058 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:36.684+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:37.058 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:36.684+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:37.058 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:36.684+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:37.058 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:36.684+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:37.057+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:37.057+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:37.057+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:37.058+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:37.111+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:37.111+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:37.111+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:37.111+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:38.004 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:37.878+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 861 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:38.004 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:37.676+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:38.004 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:37.676+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:38.004 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:37.676+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:38.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:37.676+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:38.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:38.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:38.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:38.079+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:38.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:38.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:38.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:38.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.013+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:38.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:38.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:38.979 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:38.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:38.979 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:38.645+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:38.989+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:39.124+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:39.124+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:39.124+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:39.124+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:39.946 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:39.685+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:39.947 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:39.685+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:39.947 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:39.685+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:39.947 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:39.685+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:39.946+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:39.946+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:39.946+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:39.946+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:40.121+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:40.121+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:40.121+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:40.121+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:40.638+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:40.638+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:40.638+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:40.638+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:40.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:40.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:40.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:40.963+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:41.123+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:41.123+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:41.123+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:41.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:41.123+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:41.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:41.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:41.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:41.614+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:42.152+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:42.152+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:42.152+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:42.152+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:41.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:42.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:41.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:42.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:41.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:42.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:41.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:42.994 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:42.879+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 891 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:42.995 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:42.658+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:42.995 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:42.658+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:42.995 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:42.658+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:42.995 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:42.659+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:43.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:43.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:43.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:43.110+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:42.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:42.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:42.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:42.994+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:43.646+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:43.646+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:43.646+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:43.646+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:44.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:44.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:44.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:44.110+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:43.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:43.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:43.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:43.984+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:44.605+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:44.605+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:44.605+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:44.605+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:45.140+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:45.140+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:45.140+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:45.140+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:44.954+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:44.954+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:44.954+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:45.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:44.954+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:45.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:45.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:45.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:45.569+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:46.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:46.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:46.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:46.172+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:46.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:46.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.003+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:46.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:46.555+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:46.555+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:46.555+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:46.555+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:47.216 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.957+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:47.216 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.957+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:47.216 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.957+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:47.216 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:46.957+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:47.512 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:47.215+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:47.513 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:47.215+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:47.513 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:47.215+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:47.513 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:47.215+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:47.511+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:47.511+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:47.511+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:47.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:47.511+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:47.879+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 920 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:47.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:47.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:47.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:47.988+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:48.562 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:48.229+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:48.562 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:48.229+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:48.562 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:48.229+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:48.562 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:48.229+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:48.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:48.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:48.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:48.560+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:49.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:49.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:49.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:49.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:49.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:49.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:49.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:49.003+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:49.574 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:49.254+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:49.574 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:49.254+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:49.574 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:49.254+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:49.574 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:49.254+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:49.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:49.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:49.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:49.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:49.573+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:50.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:50.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:50.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:50.045+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:50.279+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:50.279+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:50.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:50.279+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:50.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:50.279+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:50.585+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:50.585+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:50.585+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:50.585+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:51.017+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:51.017+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:51.017+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:51.018+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:51.237+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:51.237+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:51.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:51.237+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:51.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:51.238+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:51.587+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:51.587+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:51.587+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:51.587+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:52.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:52.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:52.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:52.033+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:52.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:52.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:52.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:52.274+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:52.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:52.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:52.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:52.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:52.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:52.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:52.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:52.600+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:53.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:52.881+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 948 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:53.041+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:53.041+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:53.041+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:53.041+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:53.530 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:53.280+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:53.531 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:53.280+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:53.531 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:53.280+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:53.531 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:53.280+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:53.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:53.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:53.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:53.583+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:54.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.016+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:54.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.016+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:54.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.016+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:54.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.017+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:54.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:54.308+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:54.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:54.308+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:54.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:54.308+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:54.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:54.309+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:54.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:54.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:54.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:54.617+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:55.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.974+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:55.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.974+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:55.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.974+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:55.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:54.975+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:55.572 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:55.261+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:55.572 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:55.261+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:55.572 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:55.261+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:55.572 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:55.261+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:55.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:55.571+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:55.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:55.571+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:55.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:55.571+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:55.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:55.571+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:56.256 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:55.930+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:56.256 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:55.930+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:56.256 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:55.930+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:56.256 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:55.930+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:56.605 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:56.275+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:56.605 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:56.275+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:56.605 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:56.275+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:56.605 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:56.276+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:56.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:56.604+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:56.604+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:56.604+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:56.604+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:57.305 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:56.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:57.305 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:56.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:57.305 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:56.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:57.305 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:56.944+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:57.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:57.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:57.617 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:57.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:57.617 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:57.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:57.617 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:57.304+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:57.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:57.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:57.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:57.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:57.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:57.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:57.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:57.616+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:58.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:23:57.882+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 977 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:23:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:57.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:57.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:57.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:57.972+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:58.592 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:58.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:58.592 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:58.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:58.592 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:58.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:58.592 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:58.273+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:58.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:58.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:58.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:58.592+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:23:59.270 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:58.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:59.270 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:58.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:59.270 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:58.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:23:59.271 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:23:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:23:58.989+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:59.604 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:59.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:59.604 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:59.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:59.604 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:59.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:23:59.604 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:23:59.270+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:23:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:59.603+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:59.603+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:59.603+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:23:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:23:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:23:59.603+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.034+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:00.241+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:00.241+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:00.241+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:00.242+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:00.597+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:00.597+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:00.597+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:00.597+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:01.269 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:01.269 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:01.269 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:01.269 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:00.984+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:01.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:01.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:01.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:01.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:01.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:01.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:01.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:01.268+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:01.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:01.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:01.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:01.629+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:02.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:01.959+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:02.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:01.959+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:02.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:01.959+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:02.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:01.960+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:02.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:02.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:02.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:02.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:02.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:02.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:02.590 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:02.296+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:02.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:02.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:02.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:02.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:02.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:02.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:02.884 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:02.589+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:03.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:02.882+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1008 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:02.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:02.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:02.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:02.962+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:03.332+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:03.332+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:03.332+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:03.332+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:03.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:03.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:03.560+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:03.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:03.560+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:04.319 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:03.935+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:04.319 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:03.935+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:04.319 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:03.935+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:04.320 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:03.936+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:04.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:04.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:04.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:04.528+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:04.318+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:04.318+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:04.318+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:04.319+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:05.338 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:04.957+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:05.338 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:04.957+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:05.339 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:04.957+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:05.339 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:04.958+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:05.527+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:05.527+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:05.527+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:05.527+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:05.338+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:05.338+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:05.338+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:05.338+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:06.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:05.954+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:06.332 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:05.954+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:06.332 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:05.954+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:06.332 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:05.955+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:06.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:06.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:06.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:06.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:06.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:06.516+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:06.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:06.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:06.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:06.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:06.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:06.331+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:07.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:06.976+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:07.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:06.976+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:07.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:06.976+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:07.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:06.976+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:07.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:07.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:07.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:07.543+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:07.363+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:07.363+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:07.363+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:07.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:07.363+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:08.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:07.883+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1037 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:08.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:08.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:08.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:08.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:08.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:08.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:08.026+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:08.335+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:08.335+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:08.335+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:08.336+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:08.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:08.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:08.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:08.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:08.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:08.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:08.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:08.512+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:09.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:09.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:09.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:09.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:09.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:09.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:09.317 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:09.013+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:09.316+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:09.316+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:09.316+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:09.316+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:09.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:09.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:09.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:09.523+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:10.330 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.025+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:10.330 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.025+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:10.330 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.025+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:10.330 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.025+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:10.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:10.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:10.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:10.330+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:10.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:10.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:10.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:10.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:10.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:10.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:10.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:10.487+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:11.238 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:11.238 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.986+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:11.238 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.986+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:11.238 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:10.986+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:11.238 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:24:11.237+0000 7f6019343700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T11:24:11.239 INFO:teuthology.orchestra.run.smithi100.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T11:24:11.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:11.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:11.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:11.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:11.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:11.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:11.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:11.479+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:11.611 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:11.311+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:11.611 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:11.311+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:11.611 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:11.311+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:11.611 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:11.311+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:12.065 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.mgr | length == 1'"'"'' 2024-03-24T11:24:12.321 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:11.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:12.322 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:11.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:12.322 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:11.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:12.322 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:11.939+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:12.327 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:24:12.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:12.524+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:12.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:12.524+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:12.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:12.524+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:12.582 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:12.525+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:12.582 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:12.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:12.583 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:12.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:12.583 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:12.320+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:12.583 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:12.320+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:13.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:12.884+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1078 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:12.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:12.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:12.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:12.973+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:13.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:13.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:13.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:13.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:13.570+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:13.327+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:13.327+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:13.327+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:13.593 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:13.327+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:13.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:13.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:13.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:13.961+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:14.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:14.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:14.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:14.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:14.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:14.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:14.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:14.368+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:14.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:14.618+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:14.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:14.618+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:14.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:14.618+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:14.621 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:14.618+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:15.341 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:14.998+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:15.342 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:14.998+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:15.342 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:14.998+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:15.342 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:14.998+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:15.625 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:15.341+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:15.625 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:15.341+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:15.625 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:15.341+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:15.625 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:15.341+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:15.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:15.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:15.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:15.624+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:16.306 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:15.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:16.306 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:15.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:16.306 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:15.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:16.306 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:15.953+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:16.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:16.306+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:16.306+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:16.306+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:16.306+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:16.957 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:16.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:16.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:16.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:16.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:16.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:16.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:16.673+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:17.264 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:16.956+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:17.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:16.956+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:17.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:16.956+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:17.265 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:16.957+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:17.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:17.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:17.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:17.264+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:17.985 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:17.885+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1157 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:17.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:17.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:17.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:17.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:17.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:17.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:17.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:17.724+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:17.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:17.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:17.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:17.985+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:18.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:18.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:18.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:18.216+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:18.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:18.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:18.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:18.714+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:19.261 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:18.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:19.261 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:18.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:19.261 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:18.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:19.261 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:18.995+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:19.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:19.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:19.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:19.261+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:20.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:19.728+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:20.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:19.728+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:20.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:19.728+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:20.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:19.728+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:20.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:20.040+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:20.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:20.040+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:20.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:20.040+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:20.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:20.040+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:20.309+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:20.309+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:20.309+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:20.309+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:21.056 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:20.749+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:21.057 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:20.749+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:21.057 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:20.749+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:21.057 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:20.750+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:21.326 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:21.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:21.326 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:21.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:21.326 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:21.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:21.326 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:21.056+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:21.325+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:21.325+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:21.325+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:21.325+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:22.016 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:21.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:22.016 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:21.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:22.017 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:21.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:22.017 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:21.738+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:22.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.015+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:22.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.015+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:22.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.015+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:22.329 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.016+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:22.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:22.349+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:22.349+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:22.349+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:22.349+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:23.008 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:22.886+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1237 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:22.973+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:22.736+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:22.736+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:22.736+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:23.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:22.736+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:23.532 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:23.337+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:23.532 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:23.337+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:23.532 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:23.337+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:23.533 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:23.337+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:23.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:23.762+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:23.762+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:23.762+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:23.762+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:24.353 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:23.974+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:24.353 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:23.974+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:24.353 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:23.974+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:24.353 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:23.974+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:24.352+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:24.352+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:24.352+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:24.352+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:25.015+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:25.015+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:25.015+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:25.015+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:24.792+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:24.792+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:24.792+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:24.792+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:25.372+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:25.372+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:25.372+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:25.372+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:26.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:25.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:26.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:25.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:26.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:25.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:26.006 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:25.744+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:26.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:26.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:26.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:26.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:26.005+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:26.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:26.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:26.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:26.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:26.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:26.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:26.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:26.396+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:27.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:27.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:27.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:27.006+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:26.767+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:26.767+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:26.767+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:26.767+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:27.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:27.392+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:27.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:27.392+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:27.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:27.392+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:27.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:27.392+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:27.886+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1316 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:27.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:27.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:27.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:27.815+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:28.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:28.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.038+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:28.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:28.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:28.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:28.379+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:28.988+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:28.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:28.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:28.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:29.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:28.777+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:29.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:29.427+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:29.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:29.427+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:29.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:29.427+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:29.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:29.427+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:30.048 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:29.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:30.048 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:29.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:30.049 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:29.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:30.049 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:29.784+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:30.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:30.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:30.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:30.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:30.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:30.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:30.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:30.033+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:30.800 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:30.445+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:30.800 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:30.445+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:30.800 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:30.445+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:30.800 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:30.445+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:31.051 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:30.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:31.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:30.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:31.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:30.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:31.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:30.799+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:31.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:31.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:31.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:31.050+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:31.813 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:31.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:31.813 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:31.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:31.813 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:31.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:31.813 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:31.410+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:31.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:31.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:31.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:31.812+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:32.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:32.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:32.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:32.056+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:32.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:32.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:32.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:32.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:32.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:32.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:32.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:32.410+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:33.120 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:32.887+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1410 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:33.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:32.808+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:33.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:32.808+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:33.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:32.808+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:33.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:32.808+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:33.121 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:33.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:33.121 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:33.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:33.121 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:33.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:33.121 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:33.051+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:33.795 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:33.424+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:33.795 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:33.425+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:33.795 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:33.425+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:33.795 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:33.425+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:34.069 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:33.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:34.069 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:33.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:34.069 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:33.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:34.069 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:33.794+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:34.068+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:34.068+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:34.068+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:34.069+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:34.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:34.400+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:34.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:34.400+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:34.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:34.400+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:34.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:34.401+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:35.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:34.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:34.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:34.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:34.816+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:35.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:35.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:35.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:35.064+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:35.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:35.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:35.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:35.389+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:35.838+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:35.838+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:35.838+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:35.839+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:36.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:36.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:36.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:36.047+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:36.417+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:36.417+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:36.417+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:36.417+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:36.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:36.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:36.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:36.803+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.006+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.007+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:37.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:37.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:37.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:37.390+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:38.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:37.888+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1506 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.968+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.968+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.968+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:37.969+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:37.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:38.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:37.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:38.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:37.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:38.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:37.799+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:38.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:38.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:38.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:38.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:38.374+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:38.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:38.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:38.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:38.964+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:38.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:38.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:38.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:38.841+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:39.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:39.423+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:39.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:39.423+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:39.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:39.423+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:39.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:39.424+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:39.848+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:39.848+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:39.848+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:39.849+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:39.916+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:39.916+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:39.916+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:39.916+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:40.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:40.448+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:40.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:40.448+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:40.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:40.448+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:40.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:40.448+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:40.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:40.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:40.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:40.847+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:40.918+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:40.918+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:40.918+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:40.918+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:41.803 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:41.457+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:41.804 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:41.457+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:41.804 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:41.457+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:41.804 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:41.457+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:41.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:41.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:41.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:41.803+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:41.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:41.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:41.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:41.875+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:42.482+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:42.482+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:42.482+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:42.482+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:43.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:42.889+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1601 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:42.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:42.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:42.799+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:42.799+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:42.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:42.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:42.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:42.863+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:43.771 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:43.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:43.772 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:43.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:43.772 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:43.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:43.772 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:43.506+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:43.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:43.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:43.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:43.841+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:43.770+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:43.770+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:43.770+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:43.771+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:44.821 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:44.495+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:44.821 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:44.495+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:44.821 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:44.495+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:44.821 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:44.495+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:44.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:44.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:44.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:44.837+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:44.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:44.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:44.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:44.820+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:45.836 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:45.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:45.836 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:45.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:45.836 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:45.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:45.836 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:45.522+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:45.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:45.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:45.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:45.835+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:45.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:45.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:45.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:45.859+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:46.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:46.478+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:46.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:46.478+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:46.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:46.478+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:46.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:46.478+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:46.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:46.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:46.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:46.847+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:46.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:46.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:46.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:46.876+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:47.830 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:47.496+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:47.830 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:47.496+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:47.831 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:47.496+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:47.831 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:47.497+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:47.890+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1698 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:47.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:47.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:47.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:47.830+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:47.877+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:47.877+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:47.877+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:47.877+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:48.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:48.469+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:48.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:48.469+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:48.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:48.470+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:48.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:48.470+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:48.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:48.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:48.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:48.865+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:48.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:48.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:48.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:49.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:48.875+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:49.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:49.477+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:49.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:49.477+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:49.858 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:49.477+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:49.858 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:49.477+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:49.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:49.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:49.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:49.857+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:50.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:49.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:50.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:49.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:50.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:49.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:50.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:49.906+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:50.835 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:50.515+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:50.835 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:50.515+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:50.835 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:50.515+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:50.836 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:50.516+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:50.835+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:50.835+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:50.835+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:50.835+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:50.920+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:50.920+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:50.920+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:50.921+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:51.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:51.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:51.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:51.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:51.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:51.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:51.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:51.486+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:51.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:51.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:51.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:51.958+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:51.880+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:51.880+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:52.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:51.880+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:52.147 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:51.881+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:52.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:52.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:52.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:52.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:52.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:52.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:52.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:52.443+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:53.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:52.891+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1792 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:52.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:52.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:53.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:52.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:53.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:52.857+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:53.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:53.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:53.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:53.002+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:53.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:53.488+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:53.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:53.488+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:53.887 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:53.488+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:53.887 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:53.488+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:53.885+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:53.885+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:53.885+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:53.885+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:54.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:54.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:54.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:54.049+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:54.880 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:54.516+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:54.880 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:54.516+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:54.880 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:54.516+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:54.880 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:54.516+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:54.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:54.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:54.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:55.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:54.879+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:55.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:55.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:55.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:55.056+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:55.875 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:55.545+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:55.875 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:55.545+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:55.875 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:55.545+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:55.875 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:55.545+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:55.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:55.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:55.874+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:55.875+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:56.026+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:56.026+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:56.026+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:56.026+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:56.547+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:56.547+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:56.547+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:56.547+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:56.915+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:56.915+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:56.915+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:56.915+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:57.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:57.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:57.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:57.039+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:57.530+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:57.530+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:57.530+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:57.531+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:24:57.892+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1889 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:24:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:57.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:57.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:57.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:57.952+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:58.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:58.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:58.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:58.057+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:58.517+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:58.517+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:58.517+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:58.517+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:58.962+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:58.962+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:58.962+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:24:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:24:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:24:58.962+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:24:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:59.055+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:59.055+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:59.055+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:24:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:24:59.055+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:24:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:59.526+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:59.526+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:59.526+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:24:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:24:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:24:59.527+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.003+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.003+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:00.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:00.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:00.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:00.071+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:00.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:00.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:00.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:00.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:00.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:00.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:00.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:00.548+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:01.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.991+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:01.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.991+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:01.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.991+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:01.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:00.991+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:01.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:01.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:01.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:01.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:01.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:01.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:01.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:01.097+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:01.530+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:01.530+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:01.530+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:01.531+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.005+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:02.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:02.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:02.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:02.139+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:02.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:02.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:02.507+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:02.507+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:03.158 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:02.893+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 1985 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:03.158 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.997+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:03.159 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.997+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:03.159 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.997+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:03.159 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:02.997+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:03.474 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:03.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:03.474 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:03.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:03.474 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:03.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:03.474 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:03.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:03.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:03.474+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:03.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:03.474+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:03.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:03.474+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:03.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:03.474+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:04.004+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:04.004+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:04.004+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:04.005+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:04.142+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:04.142+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:04.142+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:04.142+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:04.485+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:04.485+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:04.485+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:04.485+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.013+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:05.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:05.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:05.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:05.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:05.437+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:05.437+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:05.437+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:05.437+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.990+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.990+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.990+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:05.990+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:06.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:06.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:06.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:06.169+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:06.452+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:06.452+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:06.452+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:06.452+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:07.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:07.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:07.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:07.170+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:07.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:07.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:07.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:07.030+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:07.423+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:07.423+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:07.423+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:07.423+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:08.210 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:07.893+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2082 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:08.210 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:08.021+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:08.211 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:08.021+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:08.211 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:08.021+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:08.211 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:08.021+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:08.209+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:08.209+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:08.209+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:08.209+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:08.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:08.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:08.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:08.407+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:09.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:09.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:09.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:09.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:09.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:09.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:09.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:09.059+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:09.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:09.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:09.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:09.234+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:09.415+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:09.415+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:09.415+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:09.415+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:10.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:10.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:10.271+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:10.271+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:10.062+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:10.062+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:10.062+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:10.062+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:10.405+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:10.405+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:10.405+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:10.406+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:11.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:11.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:11.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:11.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:11.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:11.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:11.383 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:11.056+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:11.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:11.253+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:11.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:11.253+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:11.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:11.253+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:11.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:11.253+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:11.381+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:11.381+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:11.381+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:11.381+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:12.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:12.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:12.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:12.234+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:12.019+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:12.019+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:12.019+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:12.019+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:12.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:12.414+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:12.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:12.414+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:12.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:12.414+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:12.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:12.415+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:13.222 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:12.895+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2175 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:13.223 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.019+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:13.223 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.019+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:13.223 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.019+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:13.223 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.020+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:13.536 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:13.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:13.536 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:13.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:13.536 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:13.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:13.536 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:13.367+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:13.536 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:13.222+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:13.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:13.222+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:13.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:13.222+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:13.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:13.222+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:14.323 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.982+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.982+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:13.982+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:14.227+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:14.227+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:14.227+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:14.324 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:14.228+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:14.322+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:14.322+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:14.322+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:14.323+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:14.979+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:14.979+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:14.979+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:14.980+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:15.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:15.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:15.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:15.369 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:15.208+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:15.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:15.367+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:15.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:15.368+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:16.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:15.996+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:16.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:15.996+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:16.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:15.996+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:16.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:15.997+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:16.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:16.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:16.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:16.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:16.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:16.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:16.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:16.221+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:16.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:16.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:16.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:16.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:16.348+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:17.318 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:16.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:16.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:16.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:16.974+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:17.193+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:17.193+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:17.193+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:17.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:17.193+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:17.317+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:17.317+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:17.317+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:17.318+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:18.205 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:17.896+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2272 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:18.205 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:17.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:18.206 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:17.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:18.206 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:17.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:18.206 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:17.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:18.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:18.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:18.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:18.204+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:18.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:18.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:18.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:18.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:18.294+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:19.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:18.925+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:19.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:18.925+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:19.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:18.925+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:19.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:18.925+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:19.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:19.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:19.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:19.272+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:19.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:19.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:19.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:19.184+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:20.213 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:19.933+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:20.214 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:19.933+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:20.214 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:19.933+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:20.214 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:19.934+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:20.212+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:20.212+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:20.212+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:20.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:20.212+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:20.247+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:20.247+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:20.247+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:20.247+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:21.201 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:20.917+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:21.201 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:20.917+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:21.201 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:20.917+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:21.201 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:20.917+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:21.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:21.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:21.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:21.249+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:21.200+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:21.200+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:21.200+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:21.200+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:22.231 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:21.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:22.232 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:21.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:22.232 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:21.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:22.232 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:21.897+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:22.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:22.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:22.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:22.279+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:22.231+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:22.231+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:22.231+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:22.231+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:23.252 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:22.897+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2368 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:23.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:22.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:23.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:22.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:23.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:22.939+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:23.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:22.940+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:23.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:23.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:23.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:23.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:23.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:23.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:23.537 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:23.252+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:23.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:23.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:23.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:23.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:23.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:23.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:23.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:23.272+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:24.260 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:23.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:24.260 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:23.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:24.260 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:23.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:24.260 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:23.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:24.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:24.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:24.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:24.309+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:24.259+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:24.259+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:24.259+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:24.259+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:25.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:24.979+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:25.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:24.979+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:25.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:24.979+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:25.281 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:24.980+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:25.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:25.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:25.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:25.280+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:25.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:25.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:25.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:25.303+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:26.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:25.934+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:26.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:25.934+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:26.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:25.934+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:26.235 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:25.935+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:26.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:26.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:26.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:26.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:26.234+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:26.344+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:26.344+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:26.344+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:26.344+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:27.253 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:26.953+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:27.254 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:26.953+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:27.254 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:26.953+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:27.254 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:26.953+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:27.253+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:27.253+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:27.253+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:27.253+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:27.362+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:27.362+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:27.362+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:27.362+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:28.236 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:27.898+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2465 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:28.236 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:27.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:28.237 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:27.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:28.237 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:27.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:28.237 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:27.911+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:28.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:28.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:28.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:28.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:28.236+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:28.348+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:28.348+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:28.348+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:28.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:28.348+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:29.238 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:28.950+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:29.239 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:28.950+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:29.239 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:28.950+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:29.239 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:28.951+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:29.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:29.238+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:29.238+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:29.238+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:29.238+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:29.300+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:29.300+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:29.300+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:29.301+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:30.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:29.942+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:30.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:29.942+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:30.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:29.942+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:30.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:29.943+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:30.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:30.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:30.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:30.223+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:30.290+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:30.290+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:30.290+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:30.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:30.290+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:31.258 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:30.940+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:31.258 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:30.940+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:31.258 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:30.940+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:31.258 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:30.941+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:31.257+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:31.257+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:31.257+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:31.257+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:31.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:31.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:31.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:31.284+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:32.266 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:31.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:32.267 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:31.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:32.267 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:31.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:32.267 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:31.970+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:32.266+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:32.266+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:32.266+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:32.266+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:32.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:32.286+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:32.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:32.286+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:32.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:32.286+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:32.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:32.287+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:33.239 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:32.899+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2559 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:33.240 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:32.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:33.240 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:32.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:33.240 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:32.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:33.240 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:32.952+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:33.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:33.238+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:33.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:33.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:33.239+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:33.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:33.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:33.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:33.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:33.313+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:34.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:33.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:34.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:33.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:34.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:33.972+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:34.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:33.972+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:34.278 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:34.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:34.278 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:34.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:34.278 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:34.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:34.278 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:34.201+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:34.560 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:34.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:34.560 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:34.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:34.560 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:34.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:34.560 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:34.277+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.020+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:35.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:35.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:35.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:35.189+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:35.244+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:35.244+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:35.244+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:35.244+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:36.243 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:36.243 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:36.243 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.973+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:36.243 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:35.973+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:36.243 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:36.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:36.243 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:36.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:36.244 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:36.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:36.244 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:36.172+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:36.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:36.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:36.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:36.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:36.242+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:37.192 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:36.930+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:37.192 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:36.930+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:37.192 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:36.930+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:37.192 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:36.931+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:37.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:37.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:37.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:37.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:37.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:37.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:37.633 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:37.197+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:37.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:37.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:37.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:37.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:37.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:37.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:37.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:37.191+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:38.165 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:37.900+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2656 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:38.165 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:37.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:38.166 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:37.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:38.166 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:37.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:38.166 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:37.948+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:38.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:38.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:38.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:38.196+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:38.165+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:38.165+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:38.165+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:38.165+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:39.179+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:39.179+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:39.179+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:39.180+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:38.993+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:38.993+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:38.993+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:38.993+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:39.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:39.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:39.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:39.210+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:39.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:39.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:39.989+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:39.989+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:40.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:40.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:40.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:40.208+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:40.230+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:40.230+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:40.230+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:40.230+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:41.229 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:40.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:41.229 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:40.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:41.229 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:40.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:41.229 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:40.963+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:41.229+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:41.229+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:41.229+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:41.229+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:41.241+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:41.241+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:41.241+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:41.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:41.241+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:42.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:42.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:42.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:42.245+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:42.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:42.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:42.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:42.232+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:42.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:42.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:42.005+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:42.005+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:43.247 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:42.901+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2752 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:43.032+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:43.032+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:43.032+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:43.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:43.032+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:43.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:43.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:43.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:43.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:43.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:43.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:43.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:43.275+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:43.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:43.246+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:43.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:43.246+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:43.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:43.246+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:43.542 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:43.246+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:44.272 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.004+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:44.272 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.004+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:44.272 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.004+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:44.272 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.005+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:44.277+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:44.277+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:44.277+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:44.277+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:44.272+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:44.272+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:44.272+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:44.272+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:45.263 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:45.263 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:45.263 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:45.263 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:44.958+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:45.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:45.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:45.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:45.262+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:45.262+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:45.262+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:45.262+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:45.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:45.262+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:46.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:45.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:46.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:45.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:46.262 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:45.970+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:46.263 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:45.971+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:46.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:46.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:46.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:46.262+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:46.300+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:46.300+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:46.300+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:46.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:46.300+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:46.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:46.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:46.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:46.961+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:47.229+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:47.229+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:47.229+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:47.229+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:47.299+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:47.299+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:47.299+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:47.299+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:48.186 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:47.902+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2849 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:48.187 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:47.924+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:48.187 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:47.924+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:48.187 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:47.924+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:48.187 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:47.924+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:48.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:48.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:48.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:48.186+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:48.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:48.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:48.646 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:48.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:48.646 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:48.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:48.646 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:48.265+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:49.208 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:48.899+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:49.208 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:48.899+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:49.209 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:48.899+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:49.209 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:48.900+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:49.207+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:49.207+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:49.207+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:49.207+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:49.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:49.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:49.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:49.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:49.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:49.277+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:50.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:49.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:49.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:49.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:49.859+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:50.214+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:50.214+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:50.214+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:50.214+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:50.315+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:50.315+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:50.315+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:50.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:50.315+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:50.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:50.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:50.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:51.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:50.817+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:51.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:51.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:51.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:51.235+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:51.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:51.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:51.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:51.284+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:52.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:51.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:51.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:51.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:51.786+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:52.282+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:52.282+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:52.282+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:52.282+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:52.256+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:52.256+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:52.256+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:52.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:52.256+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:53.031 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:52.902+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 2943 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:53.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:52.758+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:53.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:52.758+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:53.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:52.758+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:53.032 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:52.758+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:53.260+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:53.260+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:53.260+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:53.260+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:53.247+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:53.247+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:53.247+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:53.247+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:53.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:53.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:53.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:53.807+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:54.243+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:54.243+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:54.243+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:54.244+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:54.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:54.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:54.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:54.285+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:55.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:54.833+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:54.833+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:54.833+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:54.833+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:55.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:55.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:55.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:55.248+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:55.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:55.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:55.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:55.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:55.270+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:55.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:55.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:55.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:55.850+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:56.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:56.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:56.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:56.273+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:56.278+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:56.278+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:56.278+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:56.278+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:56.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:56.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:56.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:56.809+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:57.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:57.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:57.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:57.309+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:57.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:57.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:57.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:57.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:57.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:57.327+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:58.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:25:57.904+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3040 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:25:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:57.855+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:57.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:57.856+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:58.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:57.856+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:58.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:58.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:58.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:58.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:58.347+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:58.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:58.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:58.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:58.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:58.304+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:58.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:58.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:58.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:25:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:58.827+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:59.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:59.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:59.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:25:59.348+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:59.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:59.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:59.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:25:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:25:59.264+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:59.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:59.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:59.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:25:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:25:59.790+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:00.304+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:00.304+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:00.304+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:00.304+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:00.282+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:00.282+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:00.282+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:00.283+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:00.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:00.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:00.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:00.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:01.260+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:01.260+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:01.260+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:01.260+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:01.289+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:01.289+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:01.289+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:01.290+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:01.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:01.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:01.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:01.807+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:02.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:02.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:02.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:02.268+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:02.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:02.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:02.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:02.286+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:03.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:02.905+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3136 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:02.857+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:02.857+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:02.857+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:02.858+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:03.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:03.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:03.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:03.269+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:03.280+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:03.280+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:03.540 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:03.280+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:03.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:03.280+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:04.299 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:03.901+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:04.299 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:03.901+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:04.299 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:03.901+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:04.299 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:03.901+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:04.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:04.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:04.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:04.322+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:04.298+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:04.298+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:04.298+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:04.298+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:05.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:04.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:05.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:04.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:05.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:04.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:05.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:04.898+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:05.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:05.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:05.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:05.297+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:05.339+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:05.339+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:05.339+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:05.339+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:05.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:05.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:05.865+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:05.865+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:06.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:06.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:06.296+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:06.297+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:06.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:06.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:06.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:06.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:06.294+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:07.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:06.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:07.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:06.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:07.252 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:06.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:07.253 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:06.908+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:07.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:07.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:07.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:07.252+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:07.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:07.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:07.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:07.256+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:08.236 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:07.907+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3233 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:08.236 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:07.918+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:08.236 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:07.918+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:08.237 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:07.918+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:08.237 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:07.918+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:08.235+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:08.235+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:08.235+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:08.236+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:08.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:08.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:08.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:08.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:08.257+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:09.222 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:08.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:09.222 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:08.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:09.222 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:08.963+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:09.222 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:08.964+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:09.220+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:09.220+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:09.220+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:09.220+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:09.284+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:09.284+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:09.284+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:09.284+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:10.236 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:09.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:10.236 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:09.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:10.236 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:09.983+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:10.237 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:09.983+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:10.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:10.236+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:10.236+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:10.236+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:10.236+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:10.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:10.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:10.304+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:10.304+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:11.347 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:10.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:10.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:10.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:10.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:11.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:11.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:11.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:11.348 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:11.189+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:11.346+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:11.346+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:11.346+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:11.347+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:12.313 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:12.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:12.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:12.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:12.176+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:11.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:11.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:11.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:12.314 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:11.952+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:12.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:12.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:12.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:12.313+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:13.223 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:12.908+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3327 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:13.223 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:12.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:13.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:12.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:13.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:12.952+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:13.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:12.953+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:13.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:13.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:13.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:13.350+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:13.223+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:13.223+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:13.223+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:13.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:13.223+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:14.270 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:13.995+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:14.271 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:13.995+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:14.271 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:13.995+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:14.271 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:13.995+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:14.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:14.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:14.270+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:14.270+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:14.355+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:14.355+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:14.355+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:14.356+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:15.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:15.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:15.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:15.251+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:15.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:15.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:15.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:15.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:15.034+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:15.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:15.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:15.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:15.398+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:16.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:16.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:16.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:16.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:16.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:16.260+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:16.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:16.031+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:16.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:16.031+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:16.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:16.031+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:16.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:16.031+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:16.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:16.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:16.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:16.446+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:17.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:17.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:17.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:17.271+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:17.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:17.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:17.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:17.065+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:17.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:17.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:17.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:17.409+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:18.233 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:17.909+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3424 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:18.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:18.102+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:18.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:18.102+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:18.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:18.102+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:18.234 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:18.102+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:18.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:18.233+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:18.233+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:18.233+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:18.233+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:18.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:18.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:18.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:18.418+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:19.242 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:19.151+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:19.242 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:19.151+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:19.243 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:19.151+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:19.243 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:19.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:19.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:19.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:19.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:19.242+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:19.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:19.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:19.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:19.409+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:20.460 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:20.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:20.460 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:20.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:20.461 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:20.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:20.461 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:20.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:20.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:20.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:20.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:20.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:20.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:20.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:20.461 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:20.243+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:20.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:20.460+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:20.460+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:20.460+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:20.460+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:21.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:21.168+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:21.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:21.168+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:21.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:21.168+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:21.498 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:21.168+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:21.498 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:21.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:21.498 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:21.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:21.498 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:21.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:21.498 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:21.199+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:21.496+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:21.496+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:21.496+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:21.497+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:22.481 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:22.146+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:22.481 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:22.146+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:22.482 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:22.146+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:22.482 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:22.146+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:22.482 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:22.205+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:22.482 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:22.205+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:22.482 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:22.205+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:22.482 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:22.205+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:22.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:22.480+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:22.480+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:22.480+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:22.481+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:23.240 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:22.910+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3520 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:23.241 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:23.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:23.241 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:23.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:23.241 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:23.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:23.241 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:23.098+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:23.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:23.239+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:23.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:23.240+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:23.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:23.240+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:23.541 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:23.240+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:23.542 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:23.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:23.542 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:23.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:23.542 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:23.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:23.542 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:23.448+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:24.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:24.088+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:24.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:24.088+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:24.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:24.088+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:24.233 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:24.088+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:24.491 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:24.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:24.491 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:24.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:24.491 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:24.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:24.492 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:24.232+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:24.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:24.491+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:24.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:24.491+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:24.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:24.491+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:24.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:24.491+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:25.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:25.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:25.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:25.121+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:25.245+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:25.245+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:25.245+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:25.245+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:25.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:25.503+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:25.503+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:25.503+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:25.503+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:26.281+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:26.281+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:26.281+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:26.282+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:26.103+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:26.103+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:26.103+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:26.103+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:26.784 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:26.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:26.784 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:26.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:26.784 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:26.528+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:26.785 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:26.528+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:27.518 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:27.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:27.518 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:27.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:27.518 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:27.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:27.518 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:27.150+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:27.519 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:27.234+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:27.519 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:27.234+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:27.519 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:27.234+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:27.519 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:27.234+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:27.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:27.518+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:27.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:27.518+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:27.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:27.518+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:27.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:27.518+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:28.254 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:27.911+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3617 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:28.254 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:28.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:28.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:28.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:28.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:28.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:28.255 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:28.157+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:28.545 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:28.253+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:28.545 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:28.253+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:28.545 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:28.253+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:28.545 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:28.254+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:28.544+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:28.544+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:28.544+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:28.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:28.545+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:29.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:29.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:29.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:29.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:29.211+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:29.211+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:29.211+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:29.211+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:29.856 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:29.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:29.856 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:29.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:29.856 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:29.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:29.856 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:29.537+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:30.122+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:30.122+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:30.122+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:30.122+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:30.163+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:30.163+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:30.163+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:30.163+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:30.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:30.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:30.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:30.573+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:31.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:31.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:31.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:31.136+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:31.214+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:31.214+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:31.214+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:31.214+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:31.610+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:31.610+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:31.610+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:31.610+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:32.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:32.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:32.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:32.127+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:32.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:32.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:32.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:32.242+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:32.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:32.598+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:32.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:32.598+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:32.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:32.598+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:32.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:32.598+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:33.252 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:32.911+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3711 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:33.253 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:33.090+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:33.253 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:33.090+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:33.253 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:33.090+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:33.253 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:33.090+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:33.542 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:33.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:33.542 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:33.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:33.542 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:33.251+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:33.542 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:33.251+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:33.559+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:33.559+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:33.559+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:33.559+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:34.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:34.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:34.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:34.072+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:34.288+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:34.288+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:34.288+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:34.288+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:34.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:34.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:34.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:34.523+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:35.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:35.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:35.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:35.027+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:35.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:35.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:35.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:35.252+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:35.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:35.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:35.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:35.485+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:36.009+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:36.009+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:36.009+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:36.010+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:36.234+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:36.234+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:36.234+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:36.234+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:36.521+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:36.521+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:36.521+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:36.522+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:37.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:37.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:37.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:37.031+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:37.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:37.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:37.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:37.216+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:37.477+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:37.477+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:37.477+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:37.477+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:38.194 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:37.913+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3808 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:38.195 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:38.028+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:38.195 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:38.028+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:38.195 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:38.028+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:38.195 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:38.028+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:38.492 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:38.194+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:38.493 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:38.194+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:38.493 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:38.194+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:38.493 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:38.194+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:38.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:38.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:38.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:38.492+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:39.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:39.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:39.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:39.045+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:39.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:39.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:39.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:39.175+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:39.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:39.504+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:40.106 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:39.504+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:40.106 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:39.504+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:40.107 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:39.504+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:40.074+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:40.074+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:40.074+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:40.075+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:40.149+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:40.149+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:40.149+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:40.150+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:40.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:40.483+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:40.483+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:40.483+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:40.484+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:41.106+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:41.106+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:41.106+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:41.107+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:41.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:41.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:41.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:41.131+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:41.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:41.509+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:41.509+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:41.509+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:41.510+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:42.106+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:42.106+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:42.106+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:42.106+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:42.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:42.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:42.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:42.156+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:42.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:42.488+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:42.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:42.488+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:42.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:42.488+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:42.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:42.488+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:43.183 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:42.914+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3904 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:43.183 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:43.119+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:43.184 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:43.119+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:43.184 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:43.119+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:43.184 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:43.119+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:43.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:43.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:43.515 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:43.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:43.516 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:43.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:43.516 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:43.182+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:43.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:43.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:43.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:43.515+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:44.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:44.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:44.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:44.101+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:44.136+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:44.136+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:44.136+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:44.136+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:44.543+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:44.543+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:44.543+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:44.544+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:45.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:45.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:45.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:45.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:45.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:45.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:45.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:45.145+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:45.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:45.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:45.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:45.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:45.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:45.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:45.396 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:45.094+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:45.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:45.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:45.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:45.536+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:46.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:46.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:46.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:46.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:46.173+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:46.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:46.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:46.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:46.076+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:46.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:46.511+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:46.896 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:46.511+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:46.896 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:46.511+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:46.896 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:46.512+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:47.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:47.181+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:47.181+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:47.181+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:47.082+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:47.082+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:47.082+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:47.082+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:47.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:47.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:47.536+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:47.536+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:48.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:47.915+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 3999 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:48.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:48.141+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:48.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:48.141+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:48.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:48.141+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:48.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:48.141+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:48.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:48.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:48.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:48.071+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:48.561+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:48.561+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:48.561+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:48.561+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:49.128+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:49.128+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:49.128+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:49.129+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:49.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:49.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:49.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:49.094+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:49.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:49.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:49.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:49.563+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:50.091+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:50.091+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:50.091+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:50.092+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:50.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:50.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:50.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:50.084+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:50.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:50.612+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:50.612+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:50.612+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:50.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:50.612+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:51.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:51.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:51.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:51.127+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:51.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:51.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:51.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:51.045+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:51.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:51.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:51.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:51.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:51.613+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:52.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:52.176+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:52.176+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:52.176+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:52.176+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:52.040+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:52.040+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:52.040+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:52.040+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:52.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:52.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:52.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:52.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:52.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:52.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:52.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:52.567+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:52.916+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4096 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:53.162+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:53.162+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:53.162+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:53.163+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:53.084+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:53.084+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:53.084+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:53.084+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:53.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:53.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:53.606+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:53.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:53.607+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:54.131+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:54.131+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:54.131+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:54.131+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:54.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:54.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:54.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:54.093+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:54.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:54.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:54.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:54.609+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:55.160+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:55.160+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:55.160+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:55.160+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:55.089+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:55.089+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:55.089+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:55.089+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:55.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:55.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:55.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:55.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:55.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:55.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:55.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:55.609+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:56.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:56.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:56.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:56.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:56.098+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:56.098+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:56.098+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:56.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:56.098+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:56.596+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:56.596+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:56.596+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:56.597+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:57.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:57.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:57.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:57.180+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:57.078+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:57.078+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:57.078+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:57.078+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:57.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:57.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:57.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:57.624+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:26:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:26:57.917+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4195 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:26:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:58.166+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:58.166+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:58.166+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:58.166+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:58.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:58.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:58.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:58.030+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:58.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:58.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:58.582+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:58.583+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:26:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:59.139+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:59.139+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:59.139+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:26:59.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:26:59.140+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:59.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:59.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:59.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:26:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:26:59.071+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:26:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:59.595+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:59.595+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:59.595+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:26:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:26:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:26:59.595+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:00.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:00.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:00.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:00.143+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:00.107+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:00.107+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:00.107+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:00.107+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:00.632+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:00.632+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:00.632+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:00.633+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:01.617 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:01.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:01.617 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:01.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:01.617 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:01.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:01.617 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:01.151+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:01.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:01.153+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:01.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:01.153+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:01.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:01.153+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:01.618 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:01.153+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:01.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:01.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:01.616+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:01.617+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:02.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:02.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:02.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:02.164+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:02.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:02.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:02.587 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:02.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:02.588 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:02.189+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:02.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:02.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:02.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:02.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:02.586+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:03.176 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:02.919+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4300 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:03.177 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:03.162+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:03.177 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:03.162+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:03.177 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:03.162+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:03.177 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:03.162+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:03.543 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:03.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:03.543 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:03.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:03.544 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:03.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:03.544 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:03.176+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:03.610+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:03.610+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:03.610+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:03.610+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:04.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:04.146+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:04.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:04.146+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:04.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:04.146+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:04.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:04.147+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:04.630 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:04.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:04.630 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:04.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:04.630 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:04.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:04.630 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:04.176+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:04.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:04.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:04.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:04.628+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:05.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:05.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:05.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:05.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:05.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:05.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:05.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:05.172+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:05.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:05.188+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:05.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:05.188+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:05.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:05.188+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:05.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:05.188+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:05.597+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:05.597+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:05.597+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:05.598+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:06.134+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:06.134+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:06.134+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:06.134+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:06.160+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:06.160+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:06.160+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:06.160+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:06.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:06.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:06.567+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:06.567+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:07.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:07.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:07.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:07.094+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:07.209+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:07.209+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:07.209+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:07.209+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:07.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:07.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:07.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:07.616+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:08.170 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:07.920+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4404 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:08.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:08.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:08.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:08.081+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:08.169+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:08.169+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:08.169+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:08.171 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:08.170+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:08.627+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:08.627+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:08.627+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:08.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:08.627+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:09.044+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:09.044+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:09.044+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:09.044+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:09.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:09.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:09.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:09.144+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:09.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:09.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:09.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:09.591+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:10.177+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:10.177+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:10.177+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:10.177+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:10.035+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:10.035+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:10.035+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:10.035+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:10.546+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:10.896 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:10.546+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:10.897 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:10.546+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:10.897 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:10.546+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:11.181+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:11.181+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:11.181+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:11.181+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:11.075+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:11.075+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:11.075+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:11.075+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:11.596+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:11.596+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:11.596+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:11.597+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:12.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:12.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:12.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:12.172+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:12.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:12.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:12.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:12.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:12.092+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:12.864 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:12.600+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:12.864 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:12.600+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:12.864 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:12.600+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:12.864 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:12.600+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:13.121 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:12.920+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4498 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:13.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:13.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:13.189+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:13.190+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:13.119+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:13.119+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:13.119+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:13.120+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:13.611+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:13.611+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:13.611+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:13.611+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:14.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:14.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:14.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:14.192+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:14.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:14.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:14.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:14.072+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:14.635+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:14.635+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:14.635+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:14.635+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:15.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:15.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:15.176+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:15.176+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:15.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:15.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:15.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:15.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:15.071+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:15.636+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:15.636+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:15.636+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:15.636+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:16.181+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:16.181+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:16.181+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:16.181+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:16.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:16.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:16.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:16.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:16.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:16.112+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:16.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:16.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:16.609+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:16.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:16.610+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:17.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:17.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:17.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:17.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:17.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:17.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:17.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:17.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:17.615 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:17.153+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:17.615 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:17.153+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:17.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:17.153+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:17.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:17.153+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:17.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:17.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:17.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:17.614+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:18.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:17.922+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4602 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:18.132+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:18.132+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:18.132+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:18.132+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:18.163+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:18.163+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:18.163+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:18.164+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:18.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:18.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:18.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:18.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:18.593+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:19.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:19.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:19.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:19.092+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:19.119+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:19.119+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:19.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:19.119+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:19.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:19.119+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:19.551+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:19.551+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:19.551+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:19.551+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:20.119+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:20.119+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:20.119+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:20.119+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:20.120+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:20.120+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:20.120+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:20.121+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:20.551+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:20.551+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:20.551+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:20.551+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:21.100+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:21.100+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:21.100+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:21.101+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:21.145+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:21.145+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:21.145+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:21.145+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:21.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:21.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:21.599+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:21.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:21.600+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:22.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:22.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:22.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:22.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:22.093+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:22.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:22.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:22.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:22.190+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:22.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:22.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:22.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:22.621+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:22.621+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:23.197 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:22.923+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4698 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:23.198 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:23.135+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:23.198 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:23.135+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:23.198 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:23.135+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:23.198 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:23.136+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:23.545 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:23.197+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:23.545 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:23.197+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:23.546 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:23.197+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:23.546 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:23.197+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:23.602+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:23.602+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:23.602+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:23.603+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:24.615 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:24.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:24.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:24.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:24.159+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:24.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:24.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:24.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:24.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:24.175+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:24.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:24.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:24.615+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:24.615+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:25.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:25.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:25.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:25.172+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:25.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:25.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:25.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:25.131+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:25.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:25.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:25.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:25.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:25.628+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:26.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:26.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:26.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:26.189+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:26.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:26.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:26.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:26.117+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:26.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:26.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:26.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:26.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:26.614+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:27.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:27.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:27.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:27.150+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:27.133+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:27.133+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:27.133+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:27.133+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:27.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:27.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:27.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:27.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:27.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:27.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:27.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:27.656+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:28.224 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:27.925+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4795 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:28.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:28.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:28.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:28.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:28.175+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:28.095+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:28.095+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:28.095+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:28.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:28.095+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:28.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:28.696+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:28.696+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:28.696+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:28.697+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:29.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:29.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:29.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:29.183+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:29.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:29.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:29.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:29.081+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:30.063 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:29.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:30.063 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:29.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:30.063 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:29.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:30.063 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:29.700+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:30.219+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:30.219+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:30.219+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:30.220+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:30.062+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:30.062+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:30.062+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:30.063+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:31.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:30.721+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:31.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:30.721+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:31.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:30.721+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:31.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:30.722+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:31.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:31.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:31.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:31.022+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:31.260+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:31.260+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:31.260+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:31.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:31.261+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:32.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:31.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:32.012 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:31.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:32.012 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:31.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:32.012 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:31.724+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:32.224+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:32.224+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:32.224+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:32.224+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.011+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:32.937 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:32.926+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 4893 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:32.938 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:32.686+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:32.938 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:32.686+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:32.938 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:32.686+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:32.938 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:32.687+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:33.196 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:33.196 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:33.196 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:33.196 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:32.962+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:33.546 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:33.195+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:33.546 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:33.195+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:33.546 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:33.195+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:33.546 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:33.195+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:33.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:33.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:33.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:33.680+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:34.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:34.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:34.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:34.163+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:33.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:33.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:33.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:34.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:33.955+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:34.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:34.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:34.634+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:34.634+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:34.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:34.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:34.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:34.986+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:35.140+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:35.140+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:35.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:35.140+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:35.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:35.140+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:35.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:35.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:35.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:35.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:35.614+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:36.125+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:36.125+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:36.125+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:36.125+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:35.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:35.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:35.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:35.943+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:36.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:36.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:36.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:36.587+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:37.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:37.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:37.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:37.147+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:36.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:36.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:36.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:36.945+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:37.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:37.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:37.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:37.587+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:38.178 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:37.927+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5010 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:38.179 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:37.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:38.179 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:37.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:38.179 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:37.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:38.179 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:37.991+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:38.594 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:38.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:38.594 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:38.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:38.594 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:38.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:38.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:38.178+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:38.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:38.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:38.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:38.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:38.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:38.594+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:39.212 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:38.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:39.212 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:38.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:39.212 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:38.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:39.212 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:38.956+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:39.558 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:39.211+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:39.559 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:39.211+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:39.559 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:39.211+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:39.559 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:39.212+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:39.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:39.558+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:39.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:39.558+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:39.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:39.558+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:39.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:39.558+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:40.174 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:39.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:40.174 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:39.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:40.175 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:39.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:40.175 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:39.922+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:40.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:40.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:40.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:40.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:40.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:40.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:40.550 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:40.174+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:40.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:40.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:40.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:40.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:40.549+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:40.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:40.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:40.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:40.947+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:41.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:41.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:41.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:41.127+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:41.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:41.953 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:41.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:41.953 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:41.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:41.953 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:41.556+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:41.925+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:41.925+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:41.925+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:41.925+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:42.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:42.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:42.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:42.147+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:42.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:42.575+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:42.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:42.575+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:42.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:42.575+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:42.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:42.576+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:43.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:42.929+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5122 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:42.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:42.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:42.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:42.957+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:43.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:43.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:43.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:43.123+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:43.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:43.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:43.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:43.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:43.592+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:43.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:43.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:43.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:43.963+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:44.166+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:44.166+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:44.166+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:44.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:44.167+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:44.570+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:44.570+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:44.570+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:44.570+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:44.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:44.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:44.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:44.967+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:45.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:45.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:45.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:45.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:45.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:45.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:45.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:45.174+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:45.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:45.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:45.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:45.526+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:46.215+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:46.215+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:46.215+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:46.216+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.003+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:46.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:46.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:46.535+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:46.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:46.535+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:47.216 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:47.216 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:47.217 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:47.217 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:46.958+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:47.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:47.216+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:47.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:47.216+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:47.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:47.216+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:47.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:47.216+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:47.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:47.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:47.549+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:47.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:47.549+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:48.183 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:47.930+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5218 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:48.183 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:47.979+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:48.183 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:47.979+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:48.183 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:47.979+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:48.183 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:47.980+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:48.542 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:48.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:48.542 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:48.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:48.542 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:48.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:48.543 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:48.182+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:48.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:48.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:48.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:48.542+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:49.229 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:48.978+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:49.229 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:48.978+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:49.229 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:48.978+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:49.229 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:48.978+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:49.513 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:49.228+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:49.513 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:49.228+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:49.513 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:49.228+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:49.513 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:49.228+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:49.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:49.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:49.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:49.512+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:50.265 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:49.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:50.265 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:49.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:50.265 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:49.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:50.266 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:49.993+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:50.265+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:50.265+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:50.265+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:50.265+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:50.482+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:50.482+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:50.482+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:50.483+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:51.292 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:50.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:51.292 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:50.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:51.292 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:50.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:51.292 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:50.950+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:51.292+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:51.292+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:51.292+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:51.292+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:51.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:51.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:51.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:51.435+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:52.266 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:51.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:52.266 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:51.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:52.267 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:51.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:52.267 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:51.919+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:52.264+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:52.264+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:52.264+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:52.265+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:52.453+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:52.453+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:52.453+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:52.453+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:53.058 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:52.930+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5313 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:53.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:52.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:53.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:52.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:53.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:52.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:53.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:52.887+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:53.219+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:53.219+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:53.219+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:53.219+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:53.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:53.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:53.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:53.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:53.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:53.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:53.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:53.431+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:53.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:53.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:53.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:53.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:54.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:54.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:54.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:54.407+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:54.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:54.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:54.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:54.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:54.183+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:54.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:54.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:54.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:54.849+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:55.383+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:55.383+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:55.383+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:55.383+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:55.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:55.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:55.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:55.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:55.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:55.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:55.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:55.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:55.868+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:56.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:56.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:56.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:56.397+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:56.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:56.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:56.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:56.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:56.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:56.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:56.176+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:56.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:56.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:56.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:56.884+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:57.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:57.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:57.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:57.405+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:57.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:57.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:57.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:57.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:57.165+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:58.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:27:57.933+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5408 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:27:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:57.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:57.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:57.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:57.881+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:58.450 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:58.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:58.450 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:58.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:58.450 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:58.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:58.450 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:58.175+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:58.449+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:58.449+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:58.449+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:58.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:58.449+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:27:59.208 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:58.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:59.208 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:58.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:59.208 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:58.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:27:59.208 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:58.927+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:59.207+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:59.207+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:59.207+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:27:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:27:59.208+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:27:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:59.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:59.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:59.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:27:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:27:59.427+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:00.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:00.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:00.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:00.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:00.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:00.184+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:59.970+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:59.970+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:59.970+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:00.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:27:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:27:59.970+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:00.438+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:00.438+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:00.438+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:00.439+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:00.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:00.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:00.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:00.975+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:01.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:01.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:01.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:01.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:01.148+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:01.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:01.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:01.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:01.464+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:01.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:01.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:01.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:02.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:01.963+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:02.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:02.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:02.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:02.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:02.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:02.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:02.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:02.109+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:02.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:02.503+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:02.503+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:02.503+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:02.503+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:03.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:02.934+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5504 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:02.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:02.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:02.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:03.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:02.928+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:03.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:03.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:03.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:03.138+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:03.886 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:03.493+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:03.886 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:03.493+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:03.886 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:03.493+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:03.886 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:03.493+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:03.885+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:03.885+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:03.885+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:03.885+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:04.500 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:04.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:04.500 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:04.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:04.500 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:04.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:04.500 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:04.188+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:04.499+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:04.499+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:04.499+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:04.499+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:05.180 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:04.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:05.180 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:04.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:05.180 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:04.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:05.180 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:04.922+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:05.534 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:05.179+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:05.534 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:05.179+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:05.534 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:05.179+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:05.535 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:05.180+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:05.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:05.533+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:05.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:05.533+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:05.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:05.533+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:05.883 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:05.534+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:05.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:05.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:05.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:05.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:06.569 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:06.208+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:06.570 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:06.208+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:06.570 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:06.208+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:06.570 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:06.209+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:06.844 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:06.568+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:06.844 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:06.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:06.844 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:06.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:06.844 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:06.569+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:06.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:06.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:06.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:06.843+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:07.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:07.190+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:07.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:07.190+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:07.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:07.190+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:07.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:07.190+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:07.817 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:07.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:07.817 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:07.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:07.817 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:07.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:07.817 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:07.524+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:08.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:07.935+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5599 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:08.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:07.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:08.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:07.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:08.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:07.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:08.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:07.816+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:08.503 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:08.156+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:08.503 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:08.156+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:08.503 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:08.156+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:08.503 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:08.156+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:08.796 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:08.502+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:08.796 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:08.502+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:08.796 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:08.502+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:08.796 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:08.503+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:08.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:08.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:09.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:08.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:09.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:08.795+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:09.507 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:09.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:09.507 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:09.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:09.507 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:09.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:09.507 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:09.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:09.770 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:09.506+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:09.770 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:09.506+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:09.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:09.506+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:09.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:09.507+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:10.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:09.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:09.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:09.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:09.770+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:10.470 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:10.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:10.470 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:10.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:10.470 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:10.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:10.470 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:10.171+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:10.750 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:10.470+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:10.750 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:10.470+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:10.750 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:10.470+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:10.750 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:10.470+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:10.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:10.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:10.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:10.749+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:11.475 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:11.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:11.475 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:11.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:11.475 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:11.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:11.475 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:11.179+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:11.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:11.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:11.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:11.715+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:11.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:11.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:11.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:11.474+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:12.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:12.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:12.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:12.138+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:12.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:12.481+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:12.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:12.481+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:12.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:12.481+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:12.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:12.481+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:13.063 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:12.937+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5694 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:13.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:12.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:13.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:12.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:13.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:12.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:13.063 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:12.764+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:13.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:13.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:13.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:13.149+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:13.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:13.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:13.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:13.512+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:13.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:13.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:13.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:13.758+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:14.515 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:14.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:14.515 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:14.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:14.515 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:14.154+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:14.515 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:14.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:14.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:14.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:14.514+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:14.514+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:14.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:14.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:14.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:14.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:15.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:15.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:15.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:15.111+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:15.734+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:15.734+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:15.734+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:15.734+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:15.522+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:15.522+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:15.522+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:15.522+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:16.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:16.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:16.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:16.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:16.111+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:16.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:16.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:16.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:16.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:16.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:16.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:16.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:16.474+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:16.771+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:16.771+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:16.771+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:16.771+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:17.466 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:17.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:17.467 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:17.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:17.467 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:17.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:17.467 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:17.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:17.746 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:17.466+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:17.746 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:17.466+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:17.746 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:17.466+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:17.746 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:17.466+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:18.064 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:17.938+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5790 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:18.064 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:17.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:18.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:17.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:18.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:17.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:18.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:17.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:18.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:18.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:18.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:18.188+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:18.734 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:18.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:18.734 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:18.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:18.734 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:18.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:18.734 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:18.418+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:18.733+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:18.733+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:18.733+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:18.733+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:19.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:19.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:19.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:19.403+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:19.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:19.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:19.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:19.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:19.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:19.206+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:19.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:19.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:19.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:19.715+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:20.374+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:20.374+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:20.374+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:20.374+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:20.235+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:20.235+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:20.235+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:20.235+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:20.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:20.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:20.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:20.672+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:21.630 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:21.253+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:21.630 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:21.253+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:21.630 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:21.253+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:21.630 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:21.253+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:21.630 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:21.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:21.630 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:21.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:21.631 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:21.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:21.631 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:21.350+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:21.629+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:21.629+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:21.629+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:21.629+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:22.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:22.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:22.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:22.223+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:22.396+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:22.396+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:22.396+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:22.396+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:22.940 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:22.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:22.941 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:22.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:22.941 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:22.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:22.941 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:22.645+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:23.254 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:22.939+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5886 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:23.254+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:23.254+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:23.254+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:23.254+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:23.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:23.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:23.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:23.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:23.442+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:23.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:23.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:23.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:23.657+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:24.255+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:24.255+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:24.255+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:24.255+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:24.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:24.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:24.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:24.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:24.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:24.472+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:24.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:24.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:24.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:24.652+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:25.208+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:25.208+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:25.208+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:25.208+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:25.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:25.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:25.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:25.433+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:25.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:25.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:25.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:25.666+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:26.592 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:26.253+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:26.592 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:26.253+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:26.592 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:26.253+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:26.592 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:26.253+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:26.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:26.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:26.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:26.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:26.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:26.469+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:26.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:26.470+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:26.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:26.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:26.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:26.646+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:27.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:27.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:27.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:27.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:27.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:27.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:27.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:27.445+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:27.605 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:27.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:27.605 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:27.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:27.605 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:27.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:27.605 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:27.281+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:27.604+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:27.604+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:27.604+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:27.604+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:28.281 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:27.941+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 5982 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:28.637 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:28.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:28.637 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:28.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:28.638 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:28.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:28.638 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:28.281+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:28.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:28.399+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:28.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:28.399+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:28.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:28.399+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:28.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:28.399+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:28.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:28.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:28.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:28.636+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:29.260+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:29.260+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:29.260+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:29.260+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:29.359+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:29.359+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:29.359+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:29.359+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:29.663+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:29.663+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:29.663+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:29.663+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:30.322+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:30.322+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:30.322+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:30.322+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:30.276+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:30.276+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:30.276+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:30.276+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:30.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:30.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:30.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:30.696+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:31.333+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:31.333+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:31.333+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:31.333+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:31.293+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:31.293+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:31.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:31.293+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:31.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:31.293+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:31.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:31.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:31.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:31.722+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:32.338+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:32.338+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:32.338+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:32.338+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:32.324+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:32.324+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:32.324+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:32.324+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:32.964 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:32.941+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6077 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:32.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:32.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:32.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:32.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:32.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:32.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:32.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:32.700+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:33.308+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:33.308+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:33.308+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:33.308+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:33.355+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:33.355+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:33.355+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:33.356+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:33.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:33.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:33.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:33.742+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:34.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:34.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:34.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:34.347+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:34.349+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:34.349+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:34.349+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:34.350+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:34.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:34.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:34.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:34.739+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:35.389+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:35.389+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:35.389+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:35.389+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:35.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:35.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:35.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:35.351+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:35.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:35.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:35.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:35.714+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:36.391+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:36.391+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:36.391+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:36.391+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:36.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:36.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:36.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:36.366+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:36.678+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:36.678+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:36.678+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:36.678+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:37.391+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:37.391+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:37.391+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:37.391+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:37.406+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:37.406+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:37.406+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:37.406+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:37.967 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:37.943+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6173 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:37.968 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:37.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:37.968 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:37.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:37.968 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:37.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:37.968 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:37.709+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:38.686 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:38.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:38.686 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:38.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:38.686 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:38.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:38.686 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:38.403+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:38.687 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:38.395+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:38.687 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:38.395+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:38.687 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:38.395+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:38.687 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:38.396+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:38.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:38.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:38.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:38.685+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:39.377+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:39.377+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:39.377+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:39.378+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:39.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:39.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:39.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:39.409+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:39.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:39.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:39.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:39.643+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:40.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:40.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:40.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:40.359+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:40.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:40.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:40.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:40.442+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:40.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:40.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:40.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:40.646+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:41.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:41.331+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:41.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:41.331+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:41.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:41.331+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:41.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:41.332+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:41.601 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:41.440+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:41.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:41.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:41.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:41.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:41.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:41.441+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:41.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:41.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:41.600+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:41.600+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:42.564 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:42.392+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:42.564 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:42.392+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:42.564 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:42.392+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:42.564 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:42.392+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:42.564 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:42.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:42.565 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:42.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:42.565 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:42.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:42.565 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:42.291+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:42.563+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:42.563+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:42.563+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:42.563+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:43.316 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:42.945+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6270 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:43.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:43.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:43.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:43.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:43.391+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:43.315+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:43.315+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:43.315+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:43.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:43.316+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:43.594+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:43.594+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:43.594+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:43.594+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:44.311+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:44.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:44.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:44.312+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:44.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:44.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:44.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:44.414+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:44.550+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:44.550+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:44.550+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:44.551+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:45.579 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:45.294+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:45.579 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:45.294+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:45.579 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:45.294+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:45.579 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:45.294+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:45.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:45.443+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:45.580 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:45.443+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:45.580 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:45.443+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:45.580 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:45.443+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:45.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:45.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:45.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:45.578+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:46.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:46.300+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:46.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:46.300+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:46.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:46.300+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:46.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:46.301+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:46.630 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:46.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:46.630 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:46.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:46.630 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:46.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:46.630 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:46.459+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:46.628+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:46.628+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:46.628+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:46.628+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:47.586 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:47.330+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:47.586 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:47.330+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:47.586 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:47.330+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:47.586 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:47.330+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:47.587 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:47.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:47.587 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:47.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:47.587 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:47.445+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:47.587 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:47.445+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:47.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:47.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:47.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:47.585+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:48.311 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:47.946+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6365 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:48.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:48.310+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:48.310+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:48.310+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:48.311+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:48.406+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:48.406+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:48.406+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:48.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:48.406+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:48.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:48.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:48.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:48.595+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:49.640 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:49.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:49.640 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:49.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:49.641 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:49.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:49.641 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:49.339+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:49.641 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:49.390+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:49.641 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:49.390+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:49.641 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:49.390+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:49.641 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:49.390+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:49.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:49.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:49.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:49.640+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:50.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:50.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:50.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:50.304+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:50.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:50.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:50.431+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:50.432+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:50.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:50.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:50.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:50.679+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:51.272+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:51.272+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:51.272+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:51.272+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:51.400+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:51.400+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:51.400+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:51.400+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:51.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:51.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:51.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:51.643+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:52.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:52.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:52.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:52.439+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:52.276+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:52.276+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:52.276+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:52.276+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:52.948 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:52.673+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:52.949 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:52.673+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:52.949 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:52.673+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:52.949 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:52.673+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:53.245 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:52.948+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6459 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:53.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:53.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:53.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:53.245+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:53.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:53.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:53.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:53.553 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:53.392+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:53.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:53.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:53.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:53.680+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:54.293+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:54.293+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:54.293+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:54.294+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:54.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:54.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:54.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:54.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:54.431+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:54.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:54.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:54.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:54.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:55.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:55.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:55.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:55.245+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:55.416+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:55.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:55.416+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:55.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:55.416+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:55.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:55.416+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:55.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:55.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:55.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:55.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:56.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:56.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:56.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:56.264+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:56.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:56.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:56.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:56.388+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:56.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:56.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:56.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:56.782+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:57.313+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:57.313+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:57.313+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:57.313+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:57.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:57.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:57.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:57.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:57.404+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:58.076 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:28:57.949+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6554 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:28:58.077 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:57.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:58.077 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:57.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:58.077 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:57.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:58.077 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:57.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:58.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:58.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:58.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:58.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:58.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:58.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:58.364 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:58.270+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:58.363+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:58.363+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:58.363+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:58.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:58.363+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:58.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:58.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:58.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:28:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:58.789+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:59.320+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:59.320+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:59.320+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:28:59.320+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:59.224+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:59.224+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:59.224+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:28:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:28:59.224+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:59.825+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:59.825+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:59.825+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:28:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:28:59.825+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:00.368+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:00.368+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:00.368+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:00.368+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:00.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:00.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:00.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:00.199+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:00.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:00.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:00.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:00.875+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:01.325+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:01.325+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:01.325+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:01.325+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:01.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:01.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:01.174+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:01.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:01.174+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:01.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:01.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:01.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:01.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:02.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:02.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:02.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:02.289+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:02.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:02.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:02.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:02.206+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:03.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:02.951+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6651 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:02.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:02.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:02.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:02.858+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:03.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:03.248+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:03.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:03.248+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:03.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:03.248+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:03.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:03.249+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:03.554 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:03.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:03.554 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:03.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:03.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:03.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:03.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:03.293+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:03.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:03.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:03.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:03.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:04.242+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:04.242+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:04.242+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:04.243+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:04.325+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:04.325+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:04.325+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:04.325+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:04.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:04.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:04.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:04.841+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:05.266+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:05.266+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:05.266+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:05.266+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:05.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:05.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:05.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:05.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:05.361+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:05.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:05.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:05.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:05.835+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:06.527 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:06.243+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:06.243+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:06.243+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:06.243+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:06.394+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:06.394+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:06.394+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:06.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:06.395+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:07.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:06.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:06.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:06.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:06.790+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:07.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:07.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:07.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:07.403+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:07.231+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:07.232+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:07.232+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:07.232+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:08.077 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:07.952+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6746 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:08.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:07.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:08.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:07.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:08.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:07.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:08.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:07.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:08.386 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:08.202+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:08.386 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:08.202+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:08.387 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:08.202+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:08.387 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:08.202+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:08.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:08.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:08.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:08.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:08.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:08.386+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:08.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:08.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:08.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:08.811+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:09.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:09.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:09.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:09.164+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:09.385+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:09.385+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:09.385+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:09.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:09.385+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:10.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:09.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:10.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:09.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:10.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:09.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:10.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:09.790+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:10.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:10.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:10.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:10.136+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:10.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:10.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:10.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:10.354+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:10.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:10.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:10.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:10.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:11.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:11.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:11.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:11.157+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:11.334+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:11.334+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:11.334+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:11.334+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:11.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:11.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:11.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:11.732+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:12.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:12.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:12.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:12.172+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:12.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:12.346+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:12.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:12.346+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:12.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:12.346+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:12.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:12.346+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:12.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:12.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:12.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:12.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:12.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:12.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:12.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:12.749+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:13.381 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:12.953+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6841 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:13.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:13.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:13.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:13.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:13.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:13.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:13.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:13.127+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:13.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:13.380+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:13.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:13.380+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:13.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:13.380+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:13.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:13.380+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:13.943 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:29:13.940+0000 7fd994fbf700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T11:29:13.944 INFO:teuthology.orchestra.run.smithi100.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T11:29:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:13.738+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:13.738+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:13.738+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:13.738+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:14.416 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:14.151+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:14.416 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:14.151+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:14.416 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:14.151+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:14.416 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:14.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:14.664 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c 'ceph versions | jq -e '"'"'.mgr | keys'"'"' | grep $sha1' 2024-03-24T11:29:14.692 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:14.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:14.693 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:14.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:14.693 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:14.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:14.693 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:14.415+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:14.925 INFO:teuthology.orchestra.run.smithi100.stderr:Inferring config /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/mon.smithi100/config 2024-03-24T11:29:15.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:14.699+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:15.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:14.699+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:15.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:14.699+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:15.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:14.699+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:15.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:15.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:15.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:15.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:15.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:15.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:15.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:15.184+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:15.434+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:15.434+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:15.434+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:15.434+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:16.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:15.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:16.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:15.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:16.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:15.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:16.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:15.715+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:16.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:16.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:16.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:16.148+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:16.687 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:16.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:16.687 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:16.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:16.688 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:16.403+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:16.688 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:16.403+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:16.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:16.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:16.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:16.686+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:17.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:17.387+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:17.387+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:17.387+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:17.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:17.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:17.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:17.188+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:17.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:17.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:17.956 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:17.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:17.956 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:17.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:17.956 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:17.686+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:18.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:17.953+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 6937 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:18.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:18.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:18.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:18.200+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:18.711 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:18.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:18.712 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:18.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:18.712 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:18.430+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:18.712 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:18.430+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:18.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:18.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:18.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:18.711+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:19.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:19.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:19.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:19.205+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:19.205+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:19.453+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:19.453+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:19.453+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:19.454+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:19.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:19.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:19.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:19.666+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:20.479 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:20.210+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:20.479 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:20.210+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:20.479 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:20.210+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:20.479 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:20.211+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:20.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:20.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:20.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:20.478+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:20.667+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:20.667+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:20.667+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:20.667+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:21.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:21.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:21.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:21.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:21.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:21.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:21.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:21.188+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:21.479+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:21.479+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:21.479+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:21.480+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:21.702+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:21.702+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:21.888 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:21.702+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:21.889 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:21.702+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:22.510 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:22.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:22.510 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:22.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:22.510 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:22.182+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:22.510 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:22.183+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:22.509+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:22.509+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:22.509+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:22.509+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:22.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:22.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:22.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:22.706+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:23.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:22.956+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7033 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:23.138+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:23.138+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:23.138+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:23.138+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:23.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:23.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:23.515+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:23.515+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:23.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:23.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:23.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:23.726+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:24.133+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:24.133+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:24.133+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:24.134+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:24.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:24.483+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:24.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:24.483+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:24.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:24.483+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:24.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:24.483+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:24.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:24.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:24.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:24.765+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:25.463 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:25.153+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:25.463 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:25.153+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:25.463 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:25.153+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:25.463 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:25.153+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:25.720 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:25.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:25.720 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:25.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:25.720 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:25.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:25.720 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:25.463+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:25.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:25.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:25.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:25.719+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:26.424 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:26.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:26.424 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:26.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:26.424 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:26.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:26.424 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:26.172+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:26.717 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:26.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:26.717 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:26.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:26.717 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:26.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:26.717 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:26.424+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:27.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:26.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:27.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:26.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:27.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:26.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:27.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:26.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:27.125+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:57.894055+0000 front 2024-03-24T11:18:57.895093+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:27.125+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:57.882087+0000 front 2024-03-24T11:18:57.885063+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:27.125+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:57.890215+0000 front 2024-03-24T11:18:57.910219+0000 (oldest deadline 2024-03-24T11:19:23.179691+0000) 2024-03-24T11:29:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:27.126+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:27.761 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:27.393+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:27.761 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:27.393+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:27.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:27.393+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:27.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:27.394+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:28.032 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:27.958+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7127 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:28.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:27.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:28.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:27.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:28.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:27.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:28.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:27.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:28.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:28.128+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:28.737 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:28.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:28.737 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:28.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:28.737 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:28.433+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:28.737 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:28.433+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:29.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:28.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:28.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:28.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:28.736+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:29.450 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:29.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:29.725 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:29.450+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:29.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:29.450+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:29.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:29.450+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:29.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:29.450+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:30.133 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:29.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:30.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:29.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:30.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:29.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:30.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:29.722+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:30.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:30.131+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:30.471+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:30.471+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:30.471+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:30.472+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:30.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:30.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:30.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:30.698+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:31.117+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:31.482+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:31.482+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:31.482+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:31.483+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:31.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:31.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:31.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:31.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:32.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:32.089+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:32.745 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:32.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:32.745 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:32.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:32.745 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:32.459+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:32.745 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:32.460+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:33.047 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:32.959+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7221 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:33.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:32.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.528807+0000 front 2024-03-24T11:18:59.528850+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:33.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:32.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.528824+0000 front 2024-03-24T11:18:59.528836+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:33.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:32.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.528789+0000 front 2024-03-24T11:18:59.528861+0000 (oldest deadline 2024-03-24T11:19:25.428685+0000) 2024-03-24T11:29:33.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:32.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:33.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:33.046+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:33.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:33.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:33.760 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:33.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:33.760 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:33.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:33.760 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:33.428+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:34.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:33.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:34.384 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:34.020+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:34.383+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:34.383+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:34.383+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:34.384+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:35.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:34.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:35.376 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:35.047+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:35.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:35.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:35.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:35.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:35.376+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:36.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:35.776+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:36.371 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:36.062+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:36.371+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 since back 2024-03-24T11:18:59.497962+0000 front 2024-03-24T11:18:59.497878+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:36.371+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 since back 2024-03-24T11:18:59.497905+0000 front 2024-03-24T11:18:59.497947+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:36.371+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 since back 2024-03-24T11:18:59.497925+0000 front 2024-03-24T11:18:59.497915+0000 (oldest deadline 2024-03-24T11:19:24.797795+0000) 2024-03-24T11:29:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:36.371+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:37.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:36.758+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:37.335 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:37.068+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:37.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:37.335+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:37.986 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:37.960+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7317 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:37.986 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:37.732+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:38.292 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:38.062+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:38.291+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:38.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:38.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:39.283 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:39.086+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:39.282+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:40.064 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:39.712+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:40.063+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:40.251+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:41.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:40.686+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:41.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:41.103+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:41.205+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:42.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:41.730+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:42.135+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:42.172+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:43.086 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:42.962+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7414 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:43.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:42.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:43.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:43.093+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:43.128+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:44.098 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:43.729+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:44.132+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:44.098+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:45.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:44.681+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:45.148+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:45.114+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:46.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:45.688+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:46.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:46.135+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:46.094+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:46.706+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:47.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:47.125+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:48.089 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:47.963+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7510 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:48.090 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:47.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:48.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:48.154+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:48.119+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:48.716+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:49.197+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:49.168+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:49.751+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:50.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:50.228+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:50.151+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:51.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:50.746+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:51.151+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:51.226+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:51.226+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:51.226+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:51.226+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:51.790+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:52.608 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:52.159+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:52.608 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:52.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:52.608 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:52.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:52.608 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:52.187+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:52.608 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:52.187+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:52.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:52.815+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:53.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:52.965+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7605 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:53.125+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:53.176+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:53.176+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:53.176+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:53.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:53.176+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:53.773+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:54.156+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:54.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:54.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:54.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:54.189+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:54.814+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:54.814+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:54.814+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:54.814+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:55.188+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:55.239+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:55.239+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:55.239+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:55.239+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:55.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:55.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:55.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:55.793+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:56.221+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:56.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:56.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:56.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:56.229+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:56.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:56.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:56.790+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:56.790+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:57.197+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:57.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:57.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:57.188+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:57.189+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:58.095 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:29:57.966+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7700 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:29:58.096 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:57.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:58.096 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:57.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:58.096 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:57.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:58.096 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:57.827+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:58.209+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:58.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:58.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:58.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:58.229+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:58.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:58.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:58.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:29:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:58.830+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:29:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:29:59.243+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:29:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:59.244+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:59.244+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:59.244+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:29:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:29:59.244+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:59.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:59.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:59.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:29:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:29:59.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:00.284+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:00.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:00.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:00.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:00.257+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:00.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:00.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:00.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:00.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:01.261+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:01.216+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:01.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:01.216+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:01.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:01.216+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:01.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:01.216+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:01.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:01.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:01.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:01.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:02.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:02.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:02.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:02.243+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:02.248+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:02.248+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:02.248+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:02.249+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:03.094 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:02.968+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7796 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:03.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:02.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:03.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:02.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:03.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:02.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:03.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:02.842+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:03.211+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:03.211+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:03.211+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:03.212+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:03.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:03.232+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:03.232+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:03.232+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:03.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:03.233+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:03.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:03.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:03.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:03.884+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:04.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:04.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:04.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:04.263+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:04.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:04.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:04.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:04.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:04.221+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:04.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:04.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:04.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:04.873+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:05.297+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:05.297+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:05.297+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:05.297+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:05.215+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:05.215+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:05.215+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:05.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:05.216+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:05.847+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:05.847+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:05.847+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:05.847+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:06.285+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:06.285+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:06.285+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:06.285+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:06.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:06.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:06.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:06.201+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:06.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:06.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:06.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:06.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:07.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:07.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:07.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:07.182+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:07.327+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:07.327+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:07.327+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:07.327+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:08.096 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:07.970+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7890 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:08.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:07.824+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:08.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:07.824+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:08.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:07.824+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:08.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:07.825+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:08.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:08.176+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:08.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:08.176+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:08.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:08.176+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:08.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:08.177+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:08.350+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:08.350+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:08.350+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:08.350+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:08.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:08.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:08.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:08.787+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:09.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:09.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:09.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:09.348+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:09.171+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:09.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:09.171+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:09.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:09.171+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:09.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:09.171+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:09.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:09.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:09.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:09.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:10.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:10.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:10.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:10.373+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:10.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:10.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:10.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:10.162+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:10.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:10.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:10.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:10.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:11.359+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:11.359+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:11.359+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:11.359+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:11.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:11.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:11.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:11.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:11.187+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:11.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:11.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:11.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:11.810+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:12.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:12.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:12.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:12.370+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:12.200+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:12.200+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:12.200+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:12.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:12.200+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:13.098 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:12.971+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 7985 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:13.098 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:12.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:13.098 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:12.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:13.098 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:12.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:13.098 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:12.773+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:13.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:13.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:13.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:13.170+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:13.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:13.409+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:13.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:13.409+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:13.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:13.409+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:13.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:13.410+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:14.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:13.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:14.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:13.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:14.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:13.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:14.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:13.742+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:14.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:14.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:14.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:14.135+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:14.792 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:14.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:14.792 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:14.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:14.792 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:14.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:14.793 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:14.439+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:14.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:14.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:14.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:14.792+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:15.479 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:15.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:15.479 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:15.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:15.479 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:15.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:15.479 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:15.185+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:15.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:15.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:15.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:15.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:15.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:15.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:15.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:15.479+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:15.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:15.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:15.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:15.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:16.505 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:16.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:16.505 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:16.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:16.505 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:16.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:16.505 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:16.164+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:16.504+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:16.504+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:16.504+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:16.504+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:16.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:16.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:16.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:16.735+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:17.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:17.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:17.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:17.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:17.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:17.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:17.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:17.200+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:17.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:17.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:17.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:17.487+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:17.724+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:17.724+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:17.724+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:17.725+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:18.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:17.973+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8081 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:18.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:18.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:18.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:18.156+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:18.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:18.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:18.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:18.509+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:18.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:18.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:18.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:18.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:18.749+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:19.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:19.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:19.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:19.193+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:19.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:19.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:19.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:19.522+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:19.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:19.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:19.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:19.735+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:20.532 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:20.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:20.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:20.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:20.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:20.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:20.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:20.144+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:20.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:20.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:20.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:20.532+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:20.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:20.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:20.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:20.739+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:21.549 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:21.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:21.549 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:21.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:21.549 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:21.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:21.550 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:21.179+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:21.549+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:21.549+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:21.549+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:21.549+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:21.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:21.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:21.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:21.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:21.711+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:22.508 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:22.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:22.508 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:22.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:22.508 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:22.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:22.508 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:22.169+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:22.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:22.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:22.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:22.507+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:22.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:22.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:22.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:22.735+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:23.328 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:22.974+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8177 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:23.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:23.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:23.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:23.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:23.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:23.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:23.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:23.139+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:23.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:23.517+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:23.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:23.517+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:23.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:23.517+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:23.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:23.517+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:23.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:23.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:23.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:23.711+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:24.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:24.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:24.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:24.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:24.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:24.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:24.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:24.178+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:24.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:24.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:24.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:24.532+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:24.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:24.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:24.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:24.754+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:25.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:25.180+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:25.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:25.180+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:25.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:25.180+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:25.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:25.180+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:25.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:25.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:25.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:25.538+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:25.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:25.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:25.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:25.727+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:26.516 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:26.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:26.516 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:26.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:26.516 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:26.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:26.517 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:26.188+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:26.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:26.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:26.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:26.700+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:26.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:26.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:26.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:26.516+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:27.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:27.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:27.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:27.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:27.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:27.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:27.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:27.197+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:27.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:27.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:27.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:27.540+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:27.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:27.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:27.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:27.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:27.739+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:27.976+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8272 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:28.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:28.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:28.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:28.175+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:28.690 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:28.551+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:28.690 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:28.551+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:28.691 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:28.551+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:28.691 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:28.552+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:28.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:28.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:28.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:28.690+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:29.165+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:29.165+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:29.165+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:29.166+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:29.556+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:29.556+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:29.556+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:29.556+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:29.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:29.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:29.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:29.680+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:30.583 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:30.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:30.584 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:30.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:30.584 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:30.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:30.584 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:30.161+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:30.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:30.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:30.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:30.583+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:30.631+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:30.631+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:30.631+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:30.631+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:31.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:31.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:31.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:31.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:31.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:31.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:31.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:31.152+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:31.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:31.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:31.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:31.592+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:31.649+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:31.649+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:31.649+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:31.650+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:32.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:32.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:32.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:32.141+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:32.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:32.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:32.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:32.571+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:32.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:32.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:32.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:32.612+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:32.978+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8365 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:33.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:33.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:33.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:33.192+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:33.558+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:33.558+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:33.558+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:33.558+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:33.635+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:33.635+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:33.635+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:33.635+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:34.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:34.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:34.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:34.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:34.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:34.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:34.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:34.171+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:34.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:34.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:34.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:34.645+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:34.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:34.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:34.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:34.577+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:35.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:35.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:35.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:35.141+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:35.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:35.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:35.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:35.676+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:35.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:35.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:35.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:35.622+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:36.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:36.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:36.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:36.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:36.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:36.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:36.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:36.173+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:36.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:36.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:36.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:36.613+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:36.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:36.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:36.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:36.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:36.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:36.639+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:37.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:37.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:37.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:37.152+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:37.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:37.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:37.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:37.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:37.642+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:37.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:37.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:37.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:37.981 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:37.679+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:37.978+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8461 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:38.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:38.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:38.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:38.125+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:38.627+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:38.627+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:38.627+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:38.627+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:38.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:38.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:38.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:38.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:38.725+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:39.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:39.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:39.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:39.133+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:39.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:39.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:39.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:39.637+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:39.699+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:39.699+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:39.699+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:39.699+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:40.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:40.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:40.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:40.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:40.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:40.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:40.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:40.180+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:40.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:40.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:40.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:40.594+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:40.681+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:40.681+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:40.681+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:40.681+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:41.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:41.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:41.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:41.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:41.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:41.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:41.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:41.144+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:41.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:41.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:41.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:41.594+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:41.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:41.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:41.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:41.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:42.577 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:42.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:42.577 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:42.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:42.577 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:42.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:42.577 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:42.159+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:42.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:42.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:42.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:42.577+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:42.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:42.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:42.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:42.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:42.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:43.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:42.981+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8557 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:43.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:43.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:43.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:43.179+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:43.595+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:43.596+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:43.596+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:43.596+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:43.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:43.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:43.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:43.716+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:44.628 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:44.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:44.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:44.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:44.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:44.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:44.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:44.160+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:44.628+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:44.628+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:44.628+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:44.628+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:44.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:44.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:44.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:44.715+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:45.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:45.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:45.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:45.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:45.174+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:45.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:45.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:45.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:45.658+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:45.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:45.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:45.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:45.683+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:46.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:46.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:46.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:46.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:46.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:46.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:46.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:46.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:46.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:46.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:46.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:47.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:46.676+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:47.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:47.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:47.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:47.123+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:47.984 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:47.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:47.984 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:47.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:47.984 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:47.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:47.984 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:47.666+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:47.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:47.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:47.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:47.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:47.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:47.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:47.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:47.676+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:48.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:47.983+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8653 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:48.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:48.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:48.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:48.170+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:48.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:48.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:48.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:48.633+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:48.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:48.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:48.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:48.680+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:49.168+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:49.168+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:49.168+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:49.168+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:49.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:49.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:49.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:49.678+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:49.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:49.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:49.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:49.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:50.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:50.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:50.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:50.202+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:50.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:50.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:50.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:50.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:50.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:50.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:50.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:50.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:51.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:51.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:51.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:51.173+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:51.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:51.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:51.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:52.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:51.668+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:51.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:51.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:52.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:51.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:52.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:51.746+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:52.155+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:52.155+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:52.155+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:52.155+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:52.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:52.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:52.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:52.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:52.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:52.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:52.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:52.764+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:52.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:52.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:52.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:52.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:52.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:52.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:52.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:52.658+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:53.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:52.984+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8746 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:53.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:53.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:53.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:53.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:53.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:53.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:53.160+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:53.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:53.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:53.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:53.804+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:53.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:53.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:53.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:53.609+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:54.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:54.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:54.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:54.118+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:54.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:54.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:54.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:54.589+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:54.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:54.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:54.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:54.834+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:55.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:55.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:55.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:55.094+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:55.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:55.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:55.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:55.595+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:55.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:55.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:55.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:55.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:55.831+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:56.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:56.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:56.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:56.125+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:56.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:56.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:56.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:56.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:56.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:56.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:56.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:56.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:57.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:57.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:57.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:57.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:57.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:57.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:57.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:57.126+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:57.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:56.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:57.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:56.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:57.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:56.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:57.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:56.876+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:57.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:57.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:57.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:57.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:57.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:57.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:57.843 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:57.577+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:58.113 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:30:57.986+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8842 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:30:58.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:57.842+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:58.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:57.842+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:58.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:57.842+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:58.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:57.842+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:58.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:58.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:58.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:58.164+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:58.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:58.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:58.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:30:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:58.805+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:58.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:58.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:58.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:58.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:30:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:59.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:59.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:59.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:30:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:30:59.190+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:30:59.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:59.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:59.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:59.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:59.851 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:59.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:30:59.851 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:30:59.574+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:59.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:59.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:59.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:30:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:30:59.850+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:00.570 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:00.149+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:00.570 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:00.149+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:00.570 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:00.149+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:00.570 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:00.150+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:00.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:00.569+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:00.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:00.569+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:00.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:00.569+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:00.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:00.569+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:00.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:00.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:00.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:00.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:01.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:01.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:01.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:01.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:01.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:01.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:01.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:01.176+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:01.570+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:01.570+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:01.570+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:01.570+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:01.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:01.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:01.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:01.900+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:02.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:02.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:02.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:02.131+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:02.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:02.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:02.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:02.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:02.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:02.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:02.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:02.525+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:03.122 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:02.988+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 8936 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:03.123 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:02.865+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:03.123 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:02.865+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:03.123 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:02.865+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:03.123 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:02.865+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:03.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:03.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:03.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:03.122+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:03.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:03.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:03.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:03.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:03.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:03.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:03.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:03.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:03.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:03.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:03.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:03.858+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:04.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:04.166+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:04.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:04.166+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:04.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:04.166+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:04.520 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:04.166+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:04.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:04.519+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:04.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:04.519+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:04.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:04.519+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:04.863 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:04.519+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:04.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:04.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:04.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:04.862+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:05.536 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:05.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:05.536 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:05.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:05.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:05.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:05.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:05.154+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:05.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:05.535+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:05.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:05.535+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:05.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:05.535+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:05.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:05.536+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:05.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:05.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:05.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:05.871+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:06.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:06.157+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:06.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:06.157+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:06.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:06.157+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:06.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:06.157+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:06.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:06.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:06.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:06.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:07.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:07.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:07.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:07.128+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:06.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:06.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:06.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:06.902+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:07.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:07.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:07.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:07.604+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:07.989+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9032 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:08.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:08.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:08.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:08.144+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:07.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:07.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:07.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:07.931+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:08.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:08.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:08.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:08.632+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:09.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:09.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:09.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:09.109+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:08.934+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:08.934+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:08.934+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:08.934+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:09.586+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:09.586+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:09.586+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:09.586+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:10.092+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:10.092+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:10.092+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:10.092+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:09.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:09.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:09.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:09.940+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:10.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:10.550+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:10.550+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:10.550+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:10.550+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:11.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:11.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:11.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:11.052+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:10.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:10.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:10.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:10.943+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:11.510+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:11.510+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:11.510+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:11.510+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:12.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:12.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:12.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:12.077+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:11.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:11.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:11.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:11.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:12.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:12.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:12.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:12.523+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:12.991+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9125 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:13.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:13.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:13.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:13.058+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:12.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:12.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:12.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:12.994+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:13.548+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:13.548+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:13.548+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:13.548+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:14.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:14.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:14.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:14.029+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:13.984+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:14.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:13.984+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:14.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:13.984+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:14.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:13.984+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:14.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:14.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:14.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:14.539+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:15.013+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:15.013+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:15.013+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:15.013+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:15.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:15.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:15.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:15.018+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:15.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:15.018+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:15.526+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:15.526+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:15.526+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:15.526+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.000+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:16.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:16.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:16.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:16.016+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:16.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:16.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:16.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:16.541+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:17.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:17.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:17.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:17.043+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.970+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.970+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.970+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:16.970+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:17.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:17.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:17.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:17.531+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:18.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:17.992+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9222 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:17.984+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:17.984+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:17.984+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:17.984+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:18.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:18.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:18.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:18.091+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:18.485+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:18.485+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:18.485+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:18.485+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:19.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:19.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:19.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:19.101+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:19.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:19.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:19.008+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:19.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:19.008+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:19.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:19.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:19.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:19.448+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:20.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:20.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:20.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:20.086+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:20.047+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:20.047+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:20.047+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:20.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:20.048+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:20.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:20.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:20.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:20.449+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:21.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:21.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:21.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:21.052+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.035+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:21.439+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:21.439+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:21.439+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:21.440+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:22.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:22.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:22.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:22.084+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:21.997+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:22.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:22.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:22.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:22.477+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:23.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:22.994+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9317 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:23.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:23.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:23.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:23.110+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.002+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:23.511+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:23.511+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:23.511+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:23.512+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:24.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:24.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:24.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:24.144+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:24.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:24.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:23.965+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:24.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:24.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:24.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:24.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:24.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:24.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:24.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:24.487+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:25.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:25.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:25.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:25.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:25.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:25.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:25.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:25.009+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:25.119+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:25.119+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:25.119+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:25.119+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:25.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:25.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:25.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:25.487+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:26.093+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:26.093+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:26.093+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:26.094+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:26.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:26.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:26.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:26.018+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:26.520+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:26.520+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:26.520+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:26.521+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:27.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:27.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:27.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:27.052+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.025+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.025+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.025+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.025+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:27.481+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:27.481+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:27.481+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:27.482+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:27.996+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9411 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:27.991+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:28.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:28.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:28.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:28.037+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:28.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:28.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:28.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:28.526+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:28.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:28.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:28.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:28.962+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.010+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:29.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:29.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:29.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:29.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:29.513+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:29.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:29.977+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:29.977+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:29.977+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:29.977+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:30.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:30.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:30.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:30.552+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:30.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:30.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:30.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:30.969+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:30.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:30.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:30.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:30.994+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:31.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:31.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:31.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:31.561+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:31.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:31.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:31.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:31.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:31.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:31.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:31.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:31.979+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:32.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:32.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:32.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:32.534+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:32.998+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9502 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:32.975+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:32.975+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:32.975+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:32.976+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:32.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:32.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:32.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:32.992+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:33.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:33.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:33.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:33.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:33.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:33.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:33.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:33.989+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.007+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:34.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:34.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:34.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:34.521+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:34.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:34.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:34.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:34.972+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:34.965+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:35.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:35.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:35.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:35.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:35.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:35.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:35.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:35.926+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:35.937+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:35.937+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:35.937+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:35.937+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:36.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:36.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:36.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:36.522+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:36.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:36.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:36.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:36.919+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:36.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:36.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:36.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:36.934+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:37.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:37.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:37.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:37.514+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:38.000+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9590 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:37.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:37.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:37.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:37.962+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:38.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:37.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:38.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:37.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:38.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:37.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:38.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:37.974+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:38.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:38.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:38.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:38.489+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:38.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:38.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:38.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:38.974+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:39.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:39.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:39.002+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:39.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:39.002+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:39.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:39.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:39.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:39.522+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:39.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:39.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:39.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:39.935+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:40.027+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:40.027+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:40.027+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:40.027+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:40.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:40.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:40.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:40.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:40.568+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:40.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:40.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:40.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:40.952+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:41.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:41.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:41.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:41.068+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:41.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:41.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:41.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:41.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:41.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:41.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:41.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:41.915+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:42.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:42.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:42.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:42.039+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:42.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:42.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:42.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:42.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:42.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:42.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:42.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:42.645+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:43.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:43.001+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9677 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:42.961+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:42.961+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:42.961+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:42.961+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:43.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:43.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:43.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:43.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:43.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:43.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:43.022+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:43.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:43.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:43.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:43.626+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:43.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:43.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:43.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:43.946+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:44.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:44.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:44.037+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:44.037+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:44.629+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:44.629+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:44.629+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:44.629+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:44.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:44.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:44.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:44.945+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:45.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:45.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:45.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:45.083+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:45.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:45.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:45.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:45.624+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:46.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:45.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:46.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:45.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:46.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:45.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:46.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:45.937+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:46.273 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:46.052+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:46.273 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:46.052+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:46.273 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:46.052+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:46.273 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:46.052+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:46.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:46.663+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:46.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:46.664+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:46.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:46.664+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:46.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:46.664+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:46.961+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:46.961+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:46.961+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:46.961+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.033+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:47.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:47.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:47.920 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:47.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:47.920 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:47.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:47.920 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:47.655+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:48.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:48.003+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9765 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:47.985+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:47.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:47.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:47.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:47.919+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:48.968 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:48.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:48.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:48.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:48.696+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:48.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:48.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:48.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:48.969 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:48.931+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:48.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:48.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:48.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:48.968+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:49.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:49.719+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:49.719+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:49.719+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:49.719+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:49.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:49.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:49.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:49.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:49.945+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:49.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:49.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:49.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:49.978+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:51.014 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:50.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:50.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:50.761+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:50.761+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:50.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:50.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:50.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:51.015 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:50.901+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:51.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:51.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:51.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:51.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:51.014+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:52.054 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:51.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:51.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:51.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:51.789+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:51.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:51.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:51.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:52.055 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:51.878+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:52.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:52.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:52.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:52.054+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:53.019 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:53.005+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9852 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:53.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:52.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:53.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:52.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:53.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:52.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:53.020 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:52.757+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:53.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:52.867+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:53.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:52.867+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:53.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:52.867+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:53.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:52.868+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.035+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:53.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:53.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:53.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:53.733+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:53.998+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:53.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:53.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:53.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:53.912+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:54.914+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:54.914+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:54.914+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:54.914+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:54.739+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:54.739+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:55.044 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:54.739+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:55.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:54.739+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:55.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:55.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:55.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:55.043+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:56.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:55.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:56.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:55.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:56.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:55.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:56.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:55.918+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:56.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:55.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:56.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:55.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:56.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:55.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:56.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:55.758+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.034+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:56.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:56.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:56.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:56.877+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:56.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:56.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:57.000 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:56.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:57.001 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:56.733+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:56.998+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:57.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:57.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:57.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:57.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:57.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:57.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:57.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:57.686+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:57.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:57.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:57.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:57.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:57.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:57.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:57.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:57.879+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:31:58.006+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 9945 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:31:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:57.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:57.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:57.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:57.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:58.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:58.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:58.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:58.715+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:58.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:58.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:58.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:58.844+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.000+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.000+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.000+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:31:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.000+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:31:59.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:59.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:59.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:59.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:59.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:59.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:31:59.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:31:59.668+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:31:59.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:59.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:59.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:59.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:59.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:59.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:31:59.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:31:59.855+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.955+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.955+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.955+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:31:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:31:59.957+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:00.685+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:00.685+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:00.685+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:00.685+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:00.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:00.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:00.970 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:00.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:00.971 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:00.850+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:00.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:00.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:00.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:00.969+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:01.931 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:01.697+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:01.931 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:01.697+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:01.931 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:01.697+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:01.931 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:01.698+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:01.932 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:01.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:01.932 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:01.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:01.932 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:01.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:01.932 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:01.803+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:01.930+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:01.930+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:01.930+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:01.930+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:03.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:02.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:03.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:02.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:03.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:02.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:03.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:02.796+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:03.010 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:02.689+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:02.689+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:02.689+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:02.689+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:02.908+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:02.908+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:02.908+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:03.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:02.909+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:03.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:03.009+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10040 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:03.824 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:03.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:03.824 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:03.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:03.824 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:03.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:03.824 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:03.694+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:03.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:03.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:03.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:03.823+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:03.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:03.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:04.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:03.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:04.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:03.956+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:04.743+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:04.743+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:04.743+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:04.743+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:04.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:04.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:04.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:04.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:04.781+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:04.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:04.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:04.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:04.978+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:05.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:05.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:05.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:05.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:05.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:05.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:05.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:05.765+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:05.990 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:05.723+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:05.990 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:05.723+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:05.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:05.723+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:05.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:05.724+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:05.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:05.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:05.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:05.989+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:06.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:06.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:06.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:06.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:06.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:06.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:06.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:06.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:06.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:06.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:06.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:06.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:06.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:06.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:06.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:06.689+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:06.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:06.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:06.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:06.961+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:07.960 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:07.672+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:07.960 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:07.672+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:07.960 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:07.672+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:07.960 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:07.672+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:07.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:07.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:07.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:07.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:07.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:07.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:07.961 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:07.764+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:08.011+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10136 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:07.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:07.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:07.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:07.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:08.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:08.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:08.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:08.688+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:08.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:08.757+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:08.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:08.757+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:08.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:08.757+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:08.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:08.757+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:09.008+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:09.008+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:09.008+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:09.008+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:10.034 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:09.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:09.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:09.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:09.702+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:09.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:09.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:09.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:10.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:09.807+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:10.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:10.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:10.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:10.034+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:11.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:10.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:11.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:10.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:11.073 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:10.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:11.073 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:10.682+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:11.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:10.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:11.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:10.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:11.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:10.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:11.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:10.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:11.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:11.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:11.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:11.072+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:12.075 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:11.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:11.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:11.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:11.718+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:11.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:11.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:11.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:12.076 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:11.774+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:12.074+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:12.074+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:12.074+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:12.075+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:13.013 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:12.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:13.013 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:12.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:13.013 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:12.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:13.013 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:12.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:13.014 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:12.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:13.014 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:12.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:13.014 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:12.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:13.014 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:12.726+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:13.012+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10229 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:13.120+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:13.120+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:13.120+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:13.120+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:13.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:13.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:13.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:13.721+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:13.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:13.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:13.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:13.743+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:14.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:14.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:14.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:14.140+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:14.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:14.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:14.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:14.719+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:14.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:14.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:14.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:15.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:14.739+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:15.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:15.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:15.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:15.185+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:15.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:15.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:15.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:15.768+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:15.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:15.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:15.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:15.733+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:16.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:16.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:16.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:16.152+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:17.114 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:16.747+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:17.114 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:16.747+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:17.114 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:16.747+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:17.114 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:16.747+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:17.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:16.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:17.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:16.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:17.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:16.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:17.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:16.799+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:17.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:17.113+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:17.113+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:17.113+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:17.113+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:18.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:17.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:18.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:17.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:18.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:17.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:18.015 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:17.698+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:18.016 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:17.797+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:18.016 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:17.797+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:18.016 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:17.797+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:18.016 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:17.797+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:18.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:18.014+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10325 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:18.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:18.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:18.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:18.134+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:18.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:18.699+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:18.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:18.699+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:18.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:18.699+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:18.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:18.699+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:18.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:18.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:18.756+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:18.756+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:19.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:19.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:19.175+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:19.175+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:19.757+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:19.757+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:19.757+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:19.757+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:19.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:19.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:19.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:20.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:19.701+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:20.129+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:20.129+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:20.129+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:20.129+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:21.115 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:20.672+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:21.115 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:20.672+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:21.116 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:20.672+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:21.116 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:20.672+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:21.116 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:20.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:21.116 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:20.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:21.116 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:20.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:21.116 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:20.727+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:21.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:21.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:21.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:21.115+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:21.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:21.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:21.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:21.634+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:21.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:21.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:21.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:21.700+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:21.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:21.700+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:22.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:22.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:22.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:22.080+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:22.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:22.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:22.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:22.593+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:22.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:22.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:22.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:22.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:22.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:22.668+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:23.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:23.016+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10420 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:23.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:23.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:23.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:23.076+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:23.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:23.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:23.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:23.565+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:23.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:23.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:23.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:23.706+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:24.119+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:24.119+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:24.119+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:24.119+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:24.520+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:24.520+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:24.520+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:24.520+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:24.729+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:24.729+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:24.729+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:24.729+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:25.108+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:25.108+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:25.108+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:25.108+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:25.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:25.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:25.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:25.534+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:25.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:25.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:25.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:25.741+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:26.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:26.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:26.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:26.086+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:26.697 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:26.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:26.697 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:26.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:26.697 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:26.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:26.697 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:26.567+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:27.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:26.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:27.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:26.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:27.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:26.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:27.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:26.697+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:27.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:27.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:27.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:27.094+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:27.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:27.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:27.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:27.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:27.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:27.542+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:27.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:27.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:27.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:27.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:27.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:27.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:27.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:27.693+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:28.018+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10516 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:28.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:28.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:28.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:28.071+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:28.708 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:28.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:28.708 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:28.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:28.708 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:28.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:28.708 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:28.562+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:28.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:28.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:28.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:28.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:28.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:29.280 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:29.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:29.280 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:29.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:29.280 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:29.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:29.280 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:29.036+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:29.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:29.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:29.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:29.513+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:29.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:29.702+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:29.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:29.702+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:29.702+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:29.702+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:30.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:30.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:30.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:30.010+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:30.745 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:30.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:30.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:30.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:30.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:30.495+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:30.495+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:30.495+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:30.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:30.495+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:31.021+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:31.021+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:31.021+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:31.021+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:31.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:31.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:31.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:31.504+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:31.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:31.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:31.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:31.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:32.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:32.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:32.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:32.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:32.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:32.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:32.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:32.060+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:32.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:32.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:32.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:32.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:32.516+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:33.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:32.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:33.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:32.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:33.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:32.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:33.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:32.707+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:33.020+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10609 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:33.032+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:33.032+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:33.032+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:33.032+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:33.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:33.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:33.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:33.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:33.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:33.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:33.746 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:33.478+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:34.022 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:33.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:34.022 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:33.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:34.022 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:33.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:34.022 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:33.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:34.021+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:34.021+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:34.021+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:34.021+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:34.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:34.490+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:34.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:34.490+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:34.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:34.490+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:34.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:34.490+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:35.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:34.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:35.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:34.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:35.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:34.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:35.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:34.765+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:35.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:35.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:35.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:35.027+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:35.757 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:35.530+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:35.757 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:35.530+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:35.757 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:35.530+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:35.757 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:35.530+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:36.040 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:35.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:36.040 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:35.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:36.040 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:35.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:36.040 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:35.756+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:36.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.039+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:36.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:36.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:36.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:36.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:36.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:36.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:36.743 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:36.483+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:36.991+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:36.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:36.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:36.742+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:36.742+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:37.734 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:37.451+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:37.734 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:37.451+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:37.734 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:37.451+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:37.734 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:37.452+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:38.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:37.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:37.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:37.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:37.974+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:37.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:37.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:37.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:38.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:37.732+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:38.020+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10705 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:38.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:38.485+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:38.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:38.485+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:38.780 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:38.485+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:38.780 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:38.485+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:38.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:38.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:38.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:38.981+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:38.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:38.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:38.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:38.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:39.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:39.459+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:39.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:39.459+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:39.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:39.459+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:39.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:39.459+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:40.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:40.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:40.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:40.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:40.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:40.005+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:40.146 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:39.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:40.146 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:39.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:40.146 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:39.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:40.146 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:39.812+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:40.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:40.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:40.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:40.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:40.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:40.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:40.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:40.476+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:40.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:40.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:40.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:40.785+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:41.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:41.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:41.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:41.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:41.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:41.468+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:41.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:41.468+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:41.813 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:41.468+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:41.814 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:41.468+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:41.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:41.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:41.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:41.812+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.006+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:42.802 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:42.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:42.802 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:42.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:42.802 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:42.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:42.802 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:42.489+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:43.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:43.023+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10800 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:42.971+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:42.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:42.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:42.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:42.801+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:43.829 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:43.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:43.829 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:43.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:43.829 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:43.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:43.829 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:43.491+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:43.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:43.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:43.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:43.828+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:43.941+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:43.941+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:43.941+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:43.941+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:44.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:44.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:44.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:44.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:44.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:44.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:44.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:44.502+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:45.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:44.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:44.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:44.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:44.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:44.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:44.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:44.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:44.947+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:45.769 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:45.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:45.770 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:45.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:45.770 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:45.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:45.770 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:45.503+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:45.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:45.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:45.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:45.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:45.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:45.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:45.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:45.988+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:46.775 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:46.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:46.776 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:46.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:46.776 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:46.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:46.776 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:46.505+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:47.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:46.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:47.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:46.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:47.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:46.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:47.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:46.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:47.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:47.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:47.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:47.026+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:47.775 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:47.463+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:47.776 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:47.463+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:47.776 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:47.463+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:47.776 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:47.463+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:48.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:47.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:48.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:47.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:48.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:47.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:48.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:47.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:48.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:48.025+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10896 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:48.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:48.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:48.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:48.069+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:48.753 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:48.447+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:48.753 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:48.447+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:48.753 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:48.447+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:48.753 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:48.447+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:49.051 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:48.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:49.051 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:48.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:49.051 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:48.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:49.051 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:48.752+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:49.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:49.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:49.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:49.051+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:49.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:49.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:49.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:49.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:49.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:49.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:49.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:49.479+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:49.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:49.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:49.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:49.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:50.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:50.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:50.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:50.026+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:50.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:50.475+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:50.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:50.475+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:50.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:50.475+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:50.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:50.476+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:50.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:50.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:50.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:50.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:51.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:51.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:51.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:51.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:51.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:51.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:51.859 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:51.460+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:51.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:51.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:51.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:51.858+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:51.973+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:52.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:52.489+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:52.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:52.489+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:52.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:52.489+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:52.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:52.489+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:53.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:53.027+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 10989 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:52.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:52.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:52.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:52.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:52.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:52.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:52.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:52.990+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:53.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:53.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:53.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:53.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:53.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:53.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:53.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:53.508+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:53.941+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:53.941+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:53.941+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:53.941+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:53.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:53.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:53.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:53.866+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:54.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:54.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:54.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:54.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:54.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:54.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:54.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:54.499+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:54.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:54.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:54.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:54.841+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:54.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:54.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:54.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:54.920+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:55.889 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:55.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:55.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:55.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:55.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:55.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:55.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:55.513+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:55.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:55.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:55.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:55.901+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:55.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:55.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:55.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:55.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:56.849 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:56.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:56.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:56.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:56.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:56.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:56.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:56.507+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:56.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:56.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:56.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:56.857+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:56.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:56.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:56.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:56.849+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:57.869 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:57.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:57.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:57.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:57.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:57.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:57.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:57.535+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:58.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:32:58.029+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11085 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:32:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:57.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:57.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:57.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:58.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:57.886+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:57.869+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:58.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:57.869+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:58.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:57.869+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:58.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:57.869+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:58.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:58.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:58.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:58.523+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:58.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:58.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:58.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:58.901+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:58.933+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:58.933+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:58.933+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:32:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:58.933+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:32:59.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:59.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:59.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:59.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:59.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:59.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:32:59.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:32:59.492+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:59.883+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:59.883+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:59.883+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:32:59.883+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:59.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:59.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:59.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:32:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:32:59.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:00.841 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:00.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:00.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:00.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:00.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:00.486+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:00.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:00.486+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:00.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:00.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:00.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:00.841+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:00.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:00.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:00.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:00.886+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:01.812 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:01.436+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:01.812 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:01.436+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:01.812 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:01.436+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:01.812 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:01.437+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:01.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:01.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:01.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:01.811+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:01.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:01.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:01.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:01.905+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:02.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:02.452+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:02.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:02.452+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:02.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:02.452+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:02.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:02.452+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:03.072 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:03.031+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11180 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:03.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:02.821+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:03.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:02.821+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:03.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:02.821+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:03.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:02.821+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:03.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:02.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:03.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:02.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:03.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:02.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:03.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:02.907+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:03.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:03.466+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:03.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:03.466+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:03.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:03.466+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:03.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:03.467+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:03.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:03.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:03.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:03.817+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:03.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:03.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:03.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:03.886+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:04.797 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:04.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:04.797 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:04.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:04.797 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:04.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:04.797 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:04.428+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:04.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:04.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:04.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:04.796+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:04.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:04.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:04.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:04.892+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:05.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:05.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:05.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:05.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:05.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:05.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:05.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:05.402+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:05.825+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:05.825+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:05.825+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:05.826+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:05.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:05.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:05.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:06.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:05.905+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:06.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:06.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:06.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:06.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:06.386+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:06.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:06.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:06.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:06.945+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:06.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:06.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:06.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:06.794+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:07.336+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:07.336+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:07.336+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:07.337+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:08.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:07.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:08.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:07.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:08.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:07.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:08.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:07.944+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:08.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:07.780+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:08.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:07.780+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:08.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:07.780+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:08.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:07.780+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:08.338 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:08.033+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11276 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:08.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:08.337+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:08.337+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:08.337+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:08.338+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:08.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:08.814+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:08.814+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:08.814+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:08.911+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:08.911+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:08.911+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:09.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:08.911+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:09.331+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:09.331+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:09.331+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:09.331+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:09.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:09.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:09.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:09.928+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:09.774+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:09.774+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:09.774+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:09.774+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:10.357+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:10.357+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:10.357+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:10.357+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:10.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:10.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:10.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:10.777+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:10.953+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:10.953+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:10.953+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:10.953+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:11.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:11.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:11.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:11.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:11.381+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:11.960 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:11.783+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:11.961 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:11.783+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:11.961 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:11.783+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:11.961 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:11.784+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:12.287 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:11.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:12.287 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:11.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:12.288 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:11.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:12.288 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:11.960+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:12.349+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:12.349+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:12.349+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:12.349+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:13.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:13.035+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11369 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:12.832+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:12.832+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:12.832+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:12.832+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:13.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:13.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:13.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:13.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:13.004+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:13.570 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:13.343+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:13.570 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:13.343+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:13.571 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:13.343+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:13.571 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:13.344+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:13.807+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:13.807+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:13.807+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:13.807+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:14.378 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:14.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:14.378 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:14.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:14.378 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:14.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:14.378 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:14.036+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:14.377+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:14.377+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:14.377+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:14.377+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:14.838+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:14.838+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:14.838+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:14.838+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:15.070+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:15.070+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:15.070+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:15.071+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:15.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:15.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:15.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:15.353+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:16.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:15.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:16.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:15.814+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:16.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:15.814+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:16.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:15.814+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:16.371 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:16.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:16.371 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:16.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:16.371 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:16.072+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:16.371 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:16.072+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:16.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:16.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:16.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:16.370+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:17.109 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:16.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:17.110 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:16.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:17.110 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:16.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:17.110 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:16.797+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:17.109+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:17.109+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:17.109+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:17.109+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:17.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:17.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:17.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:17.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:17.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:17.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:17.765 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:17.417+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:18.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:17.764+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:18.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:17.764+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:18.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:17.764+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:18.039 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:17.764+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:18.383 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:18.036+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11466 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:18.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:18.125+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:18.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:18.125+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:18.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:18.125+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:18.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:18.125+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:18.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:18.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:18.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:18.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:18.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:18.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:18.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:18.382+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:18.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:18.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:18.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:18.798+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:19.159+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:19.159+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:19.159+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:19.159+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:19.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:19.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:19.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:19.397+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:20.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:19.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:20.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:19.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:20.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:19.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:20.116 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:19.800+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:20.114+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:20.114+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:20.114+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:20.114+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:20.816 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:20.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:20.816 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:20.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:20.816 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:20.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:20.816 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:20.448+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:21.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:20.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:21.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:20.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:21.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:20.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:21.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:20.816+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:21.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:21.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:21.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:21.074+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:21.684 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:21.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:21.684 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:21.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:21.684 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:21.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:21.684 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:21.413+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:22.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:21.781+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:22.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:21.781+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:22.068 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:21.782+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:22.068 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:21.782+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:22.066+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:22.066+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:22.066+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:22.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:22.067+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:22.787 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:22.443+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:22.788 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:22.443+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:22.788 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:22.443+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:22.788 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:22.443+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:23.040 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:22.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:23.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:22.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:23.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:22.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:23.041 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:22.787+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:23.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:23.039+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11561 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:23.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:23.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:23.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:23.065+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:23.803 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:23.399+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:23.803 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:23.399+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:23.804 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:23.399+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:23.804 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:23.399+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:23.802+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:23.802+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:23.802+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:23.803+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:24.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:24.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:24.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:24.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:24.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:24.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:24.019+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:24.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:24.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:24.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:24.387+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:24.843+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:24.843+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:24.843+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:24.844+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:25.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:25.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:25.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:25.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:25.052+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:25.879 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:25.434+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:25.879 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:25.434+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:25.879 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:25.434+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:25.879 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:25.435+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:26.091+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:26.091+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:26.091+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:26.091+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:25.878+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:25.878+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:25.878+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:25.878+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:26.464+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:26.464+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:26.464+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:26.464+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:26.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:26.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:26.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:26.926+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:27.120+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:27.120+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:27.120+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:27.120+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:27.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:27.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:27.478+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:27.478+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:28.041+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11658 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:27.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:27.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:27.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:27.954+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:28.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:28.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:28.131+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:28.132+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:28.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:28.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:28.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:28.493+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:28.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:28.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:28.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:28.990+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:29.118+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:29.118+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:29.118+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:29.118+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:29.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:29.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:29.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:29.500+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:29.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:29.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:29.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:29.957+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:30.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:30.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:30.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:30.080+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:30.546+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:30.546+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:30.546+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:30.546+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:30.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:30.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:30.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:30.994+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:31.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:31.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:31.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:31.080+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:31.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:31.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:31.594+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:31.594+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:31.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:31.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:31.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:31.987+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:32.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:32.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:32.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:32.052+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:32.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:32.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:32.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:32.619+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:33.043+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11751 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:33.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:33.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:33.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:33.012+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:33.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:33.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:33.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:33.081+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:33.587+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:33.587+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:33.587+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:33.587+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:34.024+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:34.024+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:34.024+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:34.025+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:34.067+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:34.067+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:34.067+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:34.067+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:34.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:34.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:34.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:34.564+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:35.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:35.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:35.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:35.031+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:35.101+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:35.101+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:35.101+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:35.101+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:35.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:35.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:35.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:35.572+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:36.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:36.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:36.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:36.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:36.023+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:36.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:36.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:36.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:36.057+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:36.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:36.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:36.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:36.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:36.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:36.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:36.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:36.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:37.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:37.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:37.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:37.014+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:37.033+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:37.033+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:37.033+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:37.033+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:37.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:37.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:37.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:37.607+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:38.045+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11844 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.015+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:38.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:38.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:38.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:38.021+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:38.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:38.607+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:38.607+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:38.607+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:38.608+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:38.993+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:39.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.030+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:39.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.030+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:39.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:39.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:39.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:39.560+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:39.965+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:39.965+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:39.965+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:39.965+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:40.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:39.988+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:40.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:40.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:40.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:40.579+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:40.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:40.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:40.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:40.947+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:40.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:40.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:40.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:41.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:40.946+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:41.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:41.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:41.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:41.626+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:41.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:41.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:41.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:41.942+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:41.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:41.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:41.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:41.926+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:42.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:42.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:42.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:42.624+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:43.174 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:43.047+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 11937 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:43.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:42.895+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:43.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:42.895+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:43.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:42.895+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:43.175 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:42.895+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:43.175 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:42.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:43.175 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:42.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:43.175 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:42.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:43.175 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:42.968+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:43.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:43.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:43.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:43.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:43.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:43.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:43.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:43.579+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:43.863+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:43.863+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:43.863+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:43.864+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.010+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:44.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:44.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:44.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:44.564+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:44.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:44.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:44.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:44.900+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:44.989+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:45.515+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:45.515+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:45.515+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:45.515+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:45.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:45.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:45.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:45.955+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:45.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:45.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:45.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:45.927+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:46.510+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:46.510+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:46.510+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:46.511+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:46.899+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:46.899+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:46.899+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:46.900+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:47.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:46.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:47.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:46.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:47.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:46.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:47.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:46.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:47.869 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:47.482+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:47.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:47.482+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:47.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:47.482+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:47.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:47.482+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:48.049+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12031 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:47.898+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:47.898+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:47.898+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:47.898+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:47.869+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:47.869+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:47.869+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:47.869+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:48.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:48.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:48.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:48.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:48.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:48.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:48.823 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:48.476+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:48.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:48.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:48.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:48.914+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:48.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:48.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:48.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:48.822+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:49.817 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:49.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:49.817 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:49.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:49.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:49.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:49.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:49.503+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:49.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:49.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:49.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:49.957+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:49.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:49.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:49.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:49.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:50.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:50.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:50.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:50.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:50.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:50.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:50.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:50.540+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:50.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:50.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:50.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:50.948+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:50.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:50.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:50.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:50.850+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:51.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:51.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:51.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:51.540+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:52.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:51.944+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:52.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:51.944+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:52.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:51.944+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:52.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:51.945+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:52.224 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:51.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:52.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:51.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:52.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:51.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:52.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:51.898+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:52.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:52.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:52.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:52.542+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:53.173 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:53.051+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12124 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:52.893+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:52.893+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:52.893+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:52.893+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:52.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:52.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:52.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:53.174 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:52.906+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:53.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:53.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:53.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:53.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:53.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:53.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:53.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:53.585+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:53.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:53.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:53.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:53.934+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:53.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:53.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:53.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:53.875+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:54.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:54.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:54.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:54.633+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:54.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:54.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:54.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:54.971+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:54.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:54.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:55.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:54.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:55.297 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:54.898+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:55.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:55.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:55.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:55.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:55.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:55.636+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:55.953+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:55.953+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:55.953+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:55.954+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:55.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:55.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:55.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:55.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:56.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:56.656+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:56.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:56.656+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:56.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:56.656+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:56.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:56.656+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:56.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:56.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:56.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:56.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:56.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:56.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:56.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:56.902+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:56.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:56.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:56.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:56.991+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:57.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:57.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:57.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:57.685+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:57.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:57.685+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:57.962 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:57.685+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:57.962 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:57.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:57.962 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:57.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:57.962 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:57.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:57.962 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:57.901+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:33:58.053+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12219 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:33:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:57.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:57.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:57.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:57.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:58.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:58.932+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:58.932+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:58.932+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:58.932+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:58.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:58.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:58.722+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:33:58.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:58.722+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:33:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:58.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:58.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:58.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:33:58.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:58.891+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:59.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:59.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:59.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:33:59.890+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:59.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:59.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:59.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:33:59.880+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:59.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:59.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:59.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:00.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:33:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:33:59.724+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:00.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:00.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:00.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:00.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:00.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:00.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:00.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:00.725+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:00.891+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:00.891+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:00.891+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:00.892+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:00.924+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:00.924+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:00.924+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:00.924+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:01.955 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:01.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:01.955 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:01.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:01.955 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:01.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:01.955 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:01.852+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:01.955 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:01.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:01.955 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:01.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:01.956 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:01.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:01.956 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:01.726+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:02.328 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:01.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:02.328 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:01.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:02.329 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:01.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:02.329 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:01.954+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:02.728 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:02.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:02.728 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:02.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:02.728 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:02.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:02.728 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:02.713+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:03.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:02.816+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:03.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:02.816+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:03.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:02.816+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:03.005 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:02.817+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:03.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:03.055+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12314 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.004+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:03.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:03.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:03.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:03.698+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:03.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:03.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:03.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:03.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:03.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:03.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:03.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:03.787+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:04.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:03.961+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:04.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:04.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:04.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:04.735+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:04.758+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:04.758+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:04.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:04.758+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:04.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:04.759+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:04.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:04.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:04.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:04.929+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:05.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:05.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:05.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:05.780+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:05.801+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:05.801+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:05.801+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:05.801+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:05.972+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:05.972+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:05.972+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:05.973+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:06.766+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:06.766+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:06.766+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:06.767+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:06.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:06.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:06.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:06.808+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:06.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:06.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:06.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:06.936+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:07.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:07.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:07.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:07.834+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:07.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:07.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:08.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:07.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:08.060 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:07.910+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:08.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:07.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:08.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:07.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:08.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:07.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:08.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:07.722+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:08.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:08.057+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12408 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:08.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:08.857+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:08.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:08.857+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:08.857+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:08.858+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:08.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:08.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:08.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:08.946+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:08.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:08.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:08.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:08.979 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:08.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:08.979 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:08.716+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:09.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:09.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:09.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:09.668+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:09.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:09.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:09.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:09.811+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:09.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:09.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:09.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:09.900+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:10.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:10.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:10.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:10.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:10.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:10.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:10.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:10.764+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:10.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:10.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:10.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:10.624+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:11.875 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:11.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:11.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:11.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:11.792+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:11.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:11.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:11.624+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:11.625+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:11.848+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:11.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:11.848+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:11.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:11.848+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:11.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:11.848+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:13.060 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:12.750+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:13.060 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:12.750+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:12.750+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:12.750+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:12.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:12.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:12.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:12.668+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:12.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:13.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:12.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:13.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:12.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:13.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:12.859+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:13.059+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12501 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:13.835 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:13.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:13.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:13.734+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:13.735+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:13.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:13.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:13.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:13.836 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:13.699+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:14.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:13.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:13.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:13.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:13.835+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:14.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:14.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:14.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:14.778+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:14.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:14.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:14.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:14.664+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:14.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:14.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:14.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:14.796+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:15.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:15.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:15.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:15.815+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:15.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:15.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:15.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:15.625+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:15.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:15.626+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:15.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:15.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:15.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:15.768+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:16.644 INFO:teuthology.orchestra.run.smithi100.stderr:2024-03-24T11:34:16.641+0000 7f95125d9700 0 monclient(hunting): authenticate timed out after 300 2024-03-24T11:34:16.644 INFO:teuthology.orchestra.run.smithi100.stderr:[errno 110] RADOS timed out (error connecting to the cluster) 2024-03-24T11:34:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:16.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:16.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:16.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:16.810+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:16.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:16.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:16.591+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:16.591+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:16.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:16.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:16.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:16.810+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:17.344 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-03-24T11:34:17.345 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-24T11:34:17.581 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=c81b87511b024b4db8e1b00438d9f473 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 105, in run_tasks manager = run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 56, in task p.spawn(_run_spawned, ctx, confg, taskname) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 84, in __exit__ for result in self: File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/parallel.py", line 64, in _run_spawned mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=config) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/sequential.py", line 47, in task mgr = run_tasks.run_one_task(taskname, ctx=ctx, config=confg) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 83, in run_one_task return task(**kwargs) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1364, in shell _shell(ctx, cluster_name, remote, File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 39, in _shell return remote.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi100 with status 1: 'sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c \'ceph versions | jq -e \'"\'"\'.mgr | keys\'"\'"\' | grep $sha1\'' 2024-03-24T11:34:17.585 DEBUG:teuthology.run_tasks:Unwinding manager kclient 2024-03-24T11:34:17.599 INFO:tasks.kclient:Unmounting kernel clients... 2024-03-24T11:34:17.599 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:34:17.599 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:34:17.625 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:34:17.625 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:34:17.684 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T11:34:17.824 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:17.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:17.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:17.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:17.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:17.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:17.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:17.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:17.825 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:17.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:17.826 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:17.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:17.826 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:17.569+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:17.826 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:17.569+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:17.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:17.773+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:17.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:17.773+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:17.827 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:17.773+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:17.827 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:17.774+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:18.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:18.061+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12596 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:18.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:18.750+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:18.750+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:18.750+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:18.751+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:18.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:18.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:18.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:18.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:18.526+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:18.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:18.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:18.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:18.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:19.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:19.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:19.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:19.752+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:19.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:19.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:19.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:19.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:19.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:19.575+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:19.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:19.575+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:19.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:19.575+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:19.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:19.575+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:20.814 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:20.553+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:20.814 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:20.553+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:20.814 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:20.553+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:20.814 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:20.553+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:20.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:20.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:20.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:20.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:20.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:20.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:20.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:20.802+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:20.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:20.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:20.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:20.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:21.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:21.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:21.828 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:21.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:21.829 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:21.563+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:21.829 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:21.563+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:21.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:21.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:21.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:21.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:21.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:21.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:21.829 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:21.795+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:22.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:21.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:21.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:21.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:21.828+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:22.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:22.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:22.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:22.817+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:22.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:22.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:22.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:22.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:22.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:22.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:22.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:22.594+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:22.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:22.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:22.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:22.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:22.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:22.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:22.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:22.786+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:23.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:23.063+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12692 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:23.859 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:23.600+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:23.860 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:23.600+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:23.860 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:23.600+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:23.860 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:23.601+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:23.860 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:23.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:23.860 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:23.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:23.861 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:23.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:23.861 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:23.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:23.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:23.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:23.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:24.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:23.859+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:24.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:24.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:24.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:24.830+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:24.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:24.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:24.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:24.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:24.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:24.623+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:24.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:24.623+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:24.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:24.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:24.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:24.907+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:25.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:25.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:25.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:25.878+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:25.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:25.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:25.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:25.871+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:25.643+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:25.643+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:26.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:25.643+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:26.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:25.643+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:26.913 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:26.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:26.914 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:26.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:26.914 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:26.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:26.914 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:26.838+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:26.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:26.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:26.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:26.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:26.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:26.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:26.915 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:26.648+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:26.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:26.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:26.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:26.913+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:28.066 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:27.915+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:28.066 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:27.915+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:28.067 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:27.915+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:28.067 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:27.915+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:28.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:27.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:28.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:27.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:28.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:27.673+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:28.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:27.673+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:28.068 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:27.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:28.068 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:27.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:28.068 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:27.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:28.068 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:27.847+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:28.065+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12787 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:28.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:28.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:28.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:28.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:28.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:28.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:28.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:28.862+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:28.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:28.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:28.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:28.979 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:28.699+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:28.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:28.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:28.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:28.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:28.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:28.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:28.980 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:28.928+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:29.957 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:29.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:29.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:29.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:29.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:29.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:29.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:29.681+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:29.958 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:29.816+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:29.958 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:29.816+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:29.958 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:29.816+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:29.959 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:29.816+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:30.273 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:29.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:30.274 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:29.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:30.274 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:29.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:30.274 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:29.957+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:30.991 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:30.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:30.992 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:30.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:30.992 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:30.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:30.992 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:30.695+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:30.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:30.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:30.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:30.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:30.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:30.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:30.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:30.770+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:30.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:30.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:30.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:30.991+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:31.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:31.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:31.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:31.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:31.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:31.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:31.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:31.812+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:31.971 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:31.654+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:31.971 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:31.654+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:31.971 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:31.654+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:31.971 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:31.654+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:31.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:31.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:31.969+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:31.969+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:32.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:32.631+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:32.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:32.631+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:32.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:32.631+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:32.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:32.631+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:33.068 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:32.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:33.068 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:32.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:33.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:32.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:33.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:32.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:33.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:32.781+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:33.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:32.781+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:33.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:32.781+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:33.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:32.781+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:33.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:33.067+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12881 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:33.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:33.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:33.738+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:33.738+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:33.674+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:33.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:33.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:33.675+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:33.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:33.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:33.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:33.985+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:34.936 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:34.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:34.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:34.648+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:34.648+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:34.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:34.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:34.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:34.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:34.744+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:35.232 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:34.935+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:35.232 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:34.935+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:35.232 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:34.935+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:35.232 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:34.936+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:35.941 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:35.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:35.942 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:35.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:35.942 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:35.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:35.942 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:35.759+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:35.942 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:35.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:35.942 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:35.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:35.942 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:35.645+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:35.943 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:35.646+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:35.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:35.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:35.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:36.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:35.940+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:36.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:36.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:36.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:36.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:36.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:36.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:36.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:36.798+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:36.984 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:36.661+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:36.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:36.661+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:36.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:36.661+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:36.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:36.661+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:37.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:36.981+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:36.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:36.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:36.982+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:37.996 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:37.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:37.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:37.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:37.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:37.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:37.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:37.809+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:37.997 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:37.690+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:37.997 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:37.690+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:37.997 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:37.690+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:37.998 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:37.690+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:38.304 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:38.068+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 12976 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:38.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:37.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:38.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:37.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:38.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:37.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:38.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:37.995+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:39.018 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:38.735+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:39.018 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:38.735+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:39.018 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:38.735+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:39.018 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:38.735+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:39.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:38.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:39.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:38.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:39.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:38.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:39.019 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:38.791+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.016+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.016+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.016+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:39.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.016+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:39.993+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:40.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:39.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:40.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:39.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:40.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:39.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:40.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:39.780+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:40.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:39.785+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:40.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:39.785+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:40.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:39.785+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:40.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:39.785+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:41.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:40.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:41.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:40.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:41.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:40.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:41.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:40.815+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:41.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:40.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:40.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:40.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:40.829+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:41.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.007+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:41.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:41.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:41.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:41.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:41.986 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:41.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:41.987 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:41.868+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:41.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:41.821+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:41.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:41.821+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:41.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:41.821+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:41.987 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:41.821+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:42.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:41.986+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:43.082 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:43.071+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13071 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:43.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:42.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:43.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:42.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:43.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:42.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:43.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:42.822+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:43.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:43.015+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:43.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:43.015+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:43.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:43.015+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:43.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:43.016+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:43.084 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:42.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:43.085 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:42.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:43.085 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:42.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:43.085 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:42.908+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:43.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:43.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:43.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:43.894+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:44.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:43.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:44.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:43.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:44.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:43.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:44.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:43.850+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:44.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:44.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:44.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:44.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:44.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:44.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:44.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:44.065+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:44.919+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:44.919+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:44.919+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:44.920+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:44.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:44.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:44.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:44.868+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:45.049+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:45.050+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:45.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:45.050+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:45.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:45.050+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:45.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:45.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:45.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:45.879+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:45.908+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:45.908+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:45.908+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:45.909+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:46.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:46.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:46.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:46.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:46.010+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:46.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:46.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:46.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:46.858+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:46.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:46.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:46.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:46.916+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:47.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:47.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:47.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:47.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:47.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:47.037+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:48.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:48.074+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13165 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:47.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:47.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:47.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:47.955+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:47.872+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:47.872+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:47.872+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:48.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:47.873+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:48.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:48.014+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:48.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:48.014+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:48.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:48.014+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:48.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:48.015+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:48.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:48.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:48.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:48.947+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:49.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:48.895+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:49.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:48.895+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:49.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:48.895+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:49.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:48.895+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:49.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:49.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:49.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:49.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:49.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:49.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:49.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:49.060+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:50.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:49.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:50.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:49.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:50.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:49.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:50.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:49.958+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:50.111 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:49.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:50.111 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:49.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:50.112 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:49.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:50.112 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:49.853+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:50.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:50.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:50.110+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:50.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:50.110+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:50.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:50.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:50.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:50.845+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:50.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:50.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:50.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:50.936+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:51.146+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:51.146+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:51.146+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:51.146+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:51.952 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:51.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:51.952 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:51.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:51.952 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:51.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:51.952 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:51.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:51.951+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:51.951+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:51.951+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:51.951+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:52.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:52.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:52.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:52.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:52.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:52.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:52.156+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:53.099 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:53.076+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13259 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:53.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:52.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:53.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:52.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:53.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:52.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:53.100 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:52.983+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:53.100 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:52.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:53.100 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:52.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:53.101 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:52.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:53.101 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:52.841+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:53.161+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:53.161+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:53.161+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:53.161+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:54.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:54.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:54.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:54.010+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:53.860+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:53.860+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:53.860+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:54.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:53.861+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:54.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:54.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:54.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:54.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:54.199+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:55.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:55.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:55.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:55.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:55.029+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:55.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:54.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:55.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:54.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:55.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:54.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:55.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:54.849+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:55.248+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:55.248+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:55.248+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:55.248+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.000+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.000+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.000+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.001+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:56.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:55.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:56.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:55.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:56.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:55.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:56.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:55.859+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:56.228+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:56.228+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:56.228+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:56.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:56.228+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:56.964+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:56.816+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:56.816+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:56.816+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:56.816+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:57.207+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:57.207+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:57.207+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:57.207+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:58.078 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:57.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:58.078 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:57.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:58.078 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:57.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:58.078 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:57.952+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:58.079 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:57.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:58.079 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:57.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:58.079 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:57.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:58.079 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:57.780+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:34:58.078+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13354 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:34:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:58.193+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:58.193+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:58.193+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:58.194+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:58.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:58.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:58.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:34:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:58.969+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:34:59.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:58.793+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:59.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:58.793+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:59.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:58.793+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:34:59.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:58.794+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:34:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:59.226+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:59.226+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:59.226+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:34:59.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:34:59.227+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:59.800+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:59.800+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:59.800+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:34:59.801+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:59.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:59.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:59.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:34:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:34:59.984+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:00.206+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:00.206+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:00.206+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:00.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:00.206+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:00.787+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:00.787+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:00.787+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:00.787+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:00.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:00.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:00.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:00.972+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:01.227+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:01.227+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:01.227+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:01.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:01.228+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:01.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:01.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:01.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:01.823+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:01.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:01.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:01.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:01.931+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:02.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:02.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:02.242+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:02.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:02.242+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:03.081 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:02.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:03.082 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:02.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:03.082 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:02.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:03.082 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:02.830+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:03.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:02.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:03.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:02.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:03.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:02.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:03.083 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:02.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:03.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:03.080+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13450 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:03.246+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:03.246+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:03.246+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:03.246+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:03.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:03.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:03.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:03.948+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:04.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:03.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:04.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:03.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:04.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:03.834+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:04.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:03.834+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:04.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:04.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:04.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:04.232+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:04.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:04.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:04.945+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:04.945+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:04.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:04.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:04.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:04.814+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:05.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:05.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:05.277+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:05.277+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:05.951+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:05.951+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:05.951+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:05.952+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:05.770+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:05.770+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:05.770+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:05.770+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:06.325+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:06.325+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:06.325+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:06.325+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:06.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:06.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:06.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:07.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:06.979+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:07.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:06.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:07.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:06.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:07.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:06.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:07.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:06.789+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:07.342+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:07.342+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:07.342+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:07.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:07.342+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:08.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:07.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:08.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:07.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:08.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:07.789+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:08.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:07.789+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:08.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:07.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:08.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:07.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:08.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:07.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:08.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:07.935+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:08.082+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13545 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:08.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:08.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:08.304+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:08.304+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:08.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:08.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:08.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:08.819+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:09.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:08.955+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:09.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:08.955+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:09.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:08.955+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:09.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:08.956+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:09.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:09.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:09.321+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:09.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:09.321+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:09.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:09.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:09.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:09.849+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:09.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:09.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:09.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:09.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:10.298+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:10.299+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:10.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:10.299+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:10.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:10.299+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:10.891+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:10.891+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:10.891+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:10.891+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:11.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:10.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:11.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:10.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:11.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:10.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:11.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:10.996+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:11.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:11.257+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:11.257+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:11.257+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:11.257+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:12.038 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:11.886+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:12.039 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:11.886+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:12.039 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:11.886+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:12.039 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:11.886+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:12.276+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:12.276+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:12.276+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:12.276+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:12.038+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:12.038+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:12.038+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:12.038+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:13.143 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:13.084+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13639 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:12.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:13.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:12.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:13.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:12.858+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:13.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:12.859+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:13.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:13.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:13.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:13.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:13.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:13.049+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:13.578 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:13.307+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:13.579 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:13.307+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:13.579 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:13.307+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:13.579 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:13.307+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:13.883+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:13.883+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:13.883+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:13.883+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:14.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:14.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:14.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:14.039+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:14.285+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:14.285+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:14.285+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:14.285+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:15.168 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:15.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:15.168 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:15.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:15.168 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:15.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:15.168 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:15.084+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:15.169 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:14.896+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:15.169 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:14.896+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:15.169 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:14.896+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:15.169 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:14.896+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:15.319+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:15.319+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:15.319+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:15.319+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:16.109 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:15.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:16.109 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:15.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:16.109 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:15.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:16.110 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:15.851+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:16.357+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:16.357+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:16.357+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:16.357+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:16.108+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:16.108+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:16.108+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:16.108+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:17.122 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:16.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:17.123 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:16.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:17.123 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:16.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:17.123 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:16.852+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:17.375 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:17.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:17.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:17.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:17.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:17.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:17.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:17.121+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:17.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:17.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:17.374+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:17.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:17.374+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:18.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:17.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:18.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:17.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:18.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:17.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:18.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:17.809+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:18.347 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:18.085+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13733 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:18.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:18.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:18.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:18.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:18.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:18.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:18.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:18.088+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:18.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:18.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:18.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:18.346+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:19.071 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:18.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:19.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:18.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:19.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:18.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:19.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:18.767+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:19.340 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:19.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:19.341 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:19.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:19.341 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:19.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:19.341 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:19.071+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:19.339+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:19.339+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:19.339+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:19.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:19.339+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:20.054 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:19.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:20.057 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:19.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:20.057 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:19.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:20.057 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:19.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:20.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:20.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:20.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:20.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:20.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:20.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:20.336 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:20.053+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:20.334+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:20.334+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:20.334+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:20.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:20.334+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:21.091 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:20.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:21.092 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:20.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:21.092 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:20.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:21.092 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:20.784+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:21.370 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:21.089+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:21.370 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:21.089+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:21.370 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:21.089+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:21.370 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:21.090+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:21.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:21.369+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:21.369+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:21.369+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:21.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:21.369+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:22.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:21.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:22.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:21.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:22.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:21.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:22.049 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:21.831+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:22.350 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:22.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:22.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:22.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:22.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:22.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:22.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:22.048+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:22.350+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:22.350+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:22.350+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:22.350+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:23.102 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:23.088+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13829 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:23.103 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:22.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:23.103 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:22.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:23.103 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:22.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:23.103 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:22.847+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:23.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:23.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:23.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:23.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:23.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:23.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:23.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:23.067+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:23.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:23.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:23.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:23.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:23.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:23.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:23.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:23.396+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:24.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:24.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:24.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:24.026+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:24.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:23.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:24.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:23.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:24.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:23.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:24.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:23.802+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:24.358+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:24.358+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:24.358+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:24.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:24.359+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.003+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.003+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.003+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.003+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:24.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:25.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:24.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:25.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:24.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:25.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:24.760+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:25.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:25.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:25.389+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:25.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:25.389+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:25.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:25.795+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:25.795+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:25.795+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:25.796+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:26.753 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:26.417+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:26.754 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:26.417+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:26.754 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:26.417+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:26.754 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:26.417+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:26.753+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:26.753+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:26.753+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:26.753+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:26.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:26.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:26.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:26.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:27.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:27.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:27.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:27.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:27.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:27.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:27.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:27.442+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:28.092 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:27.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:27.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:27.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:27.964+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:27.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:27.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:27.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:28.093 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:27.769+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:28.091+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 13924 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:28.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:28.438+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:28.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:28.438+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:28.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:28.438+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:28.817 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:28.439+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:29.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:29.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:29.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:29.013+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:29.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:28.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:29.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:28.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:29.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:28.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:29.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:28.816+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:29.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:29.401+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:29.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:29.401+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:29.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:29.401+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:29.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:29.401+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:30.047+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:30.047+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:30.047+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:30.047+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:29.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:30.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:29.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:30.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:29.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:30.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:29.825+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:30.528 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:30.364+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:30.528 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:30.364+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:30.528 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:30.364+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:30.528 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:30.364+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:31.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:31.040+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:31.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:31.040+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:31.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:31.040+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:31.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:31.040+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:31.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:30.833+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:31.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:30.833+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:31.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:30.833+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:31.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:30.834+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:31.694 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:31.394+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:31.695 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:31.394+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:31.695 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:31.394+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:31.695 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:31.394+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:32.016 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:31.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:32.016 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:31.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:32.016 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:31.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:32.017 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:31.854+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.016+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:32.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:32.428+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:32.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:32.428+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:32.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:32.428+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:32.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:32.429+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:33.094 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:33.094 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:33.094 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:33.094 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:32.996+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:33.095 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:32.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:33.095 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:32.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:33.095 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:32.824+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:33.095 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:32.824+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:33.093+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14016 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:33.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:33.441+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:33.826 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:33.441+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:33.826 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:33.441+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:33.826 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:33.441+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:33.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:33.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:33.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:33.825+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:33.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:33.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:33.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:33.995+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:34.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:34.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:34.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:34.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:34.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:34.410+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:34.877 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:34.410+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:34.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:34.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:34.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:34.875+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:35.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:35.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:35.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:35.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:35.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:35.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:35.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:35.043+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:35.386+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:35.386+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:35.386+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:35.387+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.028+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.028+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.028+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.029+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:35.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:35.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:35.875+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:35.875+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:36.852 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:36.431+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:36.852 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:36.431+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:36.852 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:36.431+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:36.852 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:36.431+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:37.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:36.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:37.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:36.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:37.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:36.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:37.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:36.851+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:37.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:36.851+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:37.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:37.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:37.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:37.388+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:38.119 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:38.095+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14111 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:38.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:37.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:38.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:37.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:38.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:37.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:38.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:37.868+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:38.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:37.967+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:38.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:37.967+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:38.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:37.967+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:38.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:37.967+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:38.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:38.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:38.388+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:38.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:38.388+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:38.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:38.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:38.845+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:38.845+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:38.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:39.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:38.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:39.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:38.994+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:39.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:38.994+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:39.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:39.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:39.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:39.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:39.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:39.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:39.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:39.379+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:39.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:39.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:39.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:39.799+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:40.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:39.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:39.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:39.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:39.956+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:40.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:40.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:40.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:40.346+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:40.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:40.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:40.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:40.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:40.985+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:40.985+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:40.985+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:40.985+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:41.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:41.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:41.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:41.347+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:42.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:41.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:42.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:41.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:42.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:41.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:42.012 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:41.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:42.381 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:42.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:42.381 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:42.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:42.382 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:42.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:42.382 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:42.012+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:42.381+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:42.381+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:42.381+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:42.381+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:43.098 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:43.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:43.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:43.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:43.006+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:42.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:42.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:42.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:43.099 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:42.790+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:43.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:43.097+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14206 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:43.342+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:43.342+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:43.342+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:43.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:43.342+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:43.749+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:43.749+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:43.749+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:43.749+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:44.346 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:44.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:44.347 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:44.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:44.347 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:44.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:44.347 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:44.051+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:44.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:44.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:44.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:44.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:44.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:44.346+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:44.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:44.346+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:45.066 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:44.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:45.066 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:44.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:45.067 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:44.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:45.067 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:44.744+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:45.065+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:45.065+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:45.065+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:45.066+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:45.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:45.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:45.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:45.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:45.303+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:46.063 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:45.776+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:46.063 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:45.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:46.063 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:45.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:46.064 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:45.777+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:46.286+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:46.286+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:46.286+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:46.287+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:46.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:46.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:46.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:46.063+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:46.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:46.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:46.774+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:46.774+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:47.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:47.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:47.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:47.022+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:47.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:47.276+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:47.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:47.276+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:47.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:47.276+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:47.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:47.276+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:48.060 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:47.731+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:48.060 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:47.731+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:48.060 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:47.731+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:48.061 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:47.731+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:48.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:48.099+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14301 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:48.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:48.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:48.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:48.060+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:48.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:48.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:48.293+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:48.293+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:48.980 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:48.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:48.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:48.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:48.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:48.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:48.981 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:48.718+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:49.327 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:49.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:49.327 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:49.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:49.327 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:49.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:49.327 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:49.084+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:49.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:49.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:49.326+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:49.326+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:49.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:49.726+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:49.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:49.726+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:49.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:49.726+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:49.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:49.726+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:50.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:50.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:50.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:50.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:50.112+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:50.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:50.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:50.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:50.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:50.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:50.331+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:50.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:50.331+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:51.080 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:50.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:51.080 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:50.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:51.081 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:50.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:51.081 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:50.684+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:51.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:51.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:51.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:51.312+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:51.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:51.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:51.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:51.080+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:52.089 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:51.652+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:52.089 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:51.652+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:52.090 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:51.652+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:52.090 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:51.652+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:52.318+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:52.318+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:52.318+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:52.318+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:52.089+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:52.089+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:52.089+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:52.089+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:53.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:52.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:53.089 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:52.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:53.089 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:52.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:53.089 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:52.686+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:53.351 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:53.102+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14395 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:53.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:53.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:53.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:53.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:53.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:53.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:53.352 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:53.088+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:53.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:53.350+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:53.350+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:53.350+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:53.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:53.351+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:54.125 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:53.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:54.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:53.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:54.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:53.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:54.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:53.673+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:54.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:54.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:54.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:54.125+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:54.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:54.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:54.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:54.368+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:55.104 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:54.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:55.104 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:54.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:55.104 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:54.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:55.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:54.647+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:55.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:55.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:55.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:55.126+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:55.324+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:55.324+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:55.324+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:55.324+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:55.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:55.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:55.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:55.642+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:56.146+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:56.146+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:56.146+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:56.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:56.146+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:56.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:56.274+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:56.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:56.274+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:56.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:56.274+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:56.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:56.274+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:57.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:56.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:57.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:56.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:57.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:56.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:57.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:56.687+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:57.311+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:57.311+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:57.311+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:57.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:57.311+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:57.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:57.142+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:57.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:57.142+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:57.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:57.142+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:57.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:57.142+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:58.107 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:57.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:58.108 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:57.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:58.108 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:57.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:58.108 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:57.717+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:35:58.103+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14489 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:35:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:58.111+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:58.111+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:58.111+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:58.112+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:58.314+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:58.314+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:58.314+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:58.314+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:59.117 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:58.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:59.117 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:58.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:59.117 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:58.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:35:59.117 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:58.744+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:35:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:59.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:59.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:59.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:35:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:35:59.116+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:35:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:59.291+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:59.291+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:59.291+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:35:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:35:59.291+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:00.100 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:59.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:00.100 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:59.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:00.100 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:59.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:00.101 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:35:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:35:59.695+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:00.099+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:00.099+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:00.099+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:00.099+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:00.319+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:00.319+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:00.319+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:00.319+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:01.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:00.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:01.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:00.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:01.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:00.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:01.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:00.649+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:01.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:01.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:01.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:01.141+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:01.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:01.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:01.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:01.330+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:01.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:01.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:01.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:01.636+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:02.631 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:02.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:02.632 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:02.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:02.632 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:02.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:02.632 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:02.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:02.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:02.364+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:02.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:02.364+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:02.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:02.364+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:02.633 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:02.364+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:02.631+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:02.631+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:02.631+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:02.631+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:03.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:03.105+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14576 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:03.194+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:03.194+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:03.194+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:03.195+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:03.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:03.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:03.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:03.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:03.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:03.347+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:03.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:03.347+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:03.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:03.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:03.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:03.679+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:04.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:04.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:04.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:04.165+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:04.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:04.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:04.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:04.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:04.330+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:04.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:04.331+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:04.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:04.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:04.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:04.658+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:05.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:05.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:05.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:05.206+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:05.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:05.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:05.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:05.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:05.379+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:05.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:05.380+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:05.663+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:05.663+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:05.663+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:05.663+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:06.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:06.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:06.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:06.210+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:06.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:06.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:06.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:06.395+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:06.395+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:06.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:06.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:06.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:06.645+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:07.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:07.426+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:07.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:07.426+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:07.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:07.426+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:07.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:07.427+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:07.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:07.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:07.621 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:07.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:07.621 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:07.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:07.621 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:07.249+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:07.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:07.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:07.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:07.619+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:08.109+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14659 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:08.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:08.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:08.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:08.259+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:08.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:08.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:08.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:08.576+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:08.441+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:08.441+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:08.441+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:08.442+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:09.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:09.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:09.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:09.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:09.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:09.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:09.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:09.271+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:09.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:09.419+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:09.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:09.419+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:09.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:09.419+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:09.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:09.419+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:09.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:09.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:09.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:09.617+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:10.456 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:10.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:10.457 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:10.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:10.457 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:10.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:10.457 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:10.309+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:10.588+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:10.588+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:10.588+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:10.589+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:10.456+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:10.456+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:10.456+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:10.456+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:11.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:11.409+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:11.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:11.409+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:11.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:11.409+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:11.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:11.409+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:11.600 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:11.295+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:11.600 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:11.295+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:11.600 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:11.295+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:11.601 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:11.295+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:11.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:11.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:11.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:11.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:12.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:12.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:12.569 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:12.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:12.569 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:12.269+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:12.569 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:12.269+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:12.569 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:12.391+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:12.569 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:12.391+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:12.570 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:12.391+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:12.570 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:12.391+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:12.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:12.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:12.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:12.568+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:13.111+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14741 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:13.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:13.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:13.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:13.250+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:13.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:13.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:13.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:13.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:13.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:13.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:13.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:13.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:13.407+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:14.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:14.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:14.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:14.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:14.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:14.201+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:14.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:14.202+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:14.575 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:14.372+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:14.575 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:14.372+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:14.575 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:14.372+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:14.575 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:14.372+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:14.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:14.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:14.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:14.573+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:15.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:15.238+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:15.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:15.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:15.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:15.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:15.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:15.239+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:15.569 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:15.343+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:15.569 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:15.343+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:15.569 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:15.343+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:15.569 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:15.343+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:15.567+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:15.567+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:15.567+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:15.567+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:16.532 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:16.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:16.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:16.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:16.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:16.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:16.533 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:16.239+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:16.533 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:16.329+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:16.533 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:16.329+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:16.534 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:16.329+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:16.534 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:16.329+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:16.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:16.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:16.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:16.532+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:17.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:17.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:17.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:17.259+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:17.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:17.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:17.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:17.501+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:17.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:17.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:17.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:17.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:17.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:17.312+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:17.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:17.313+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:17.762 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T11:36:17.763 WARNING:tasks.kclient:Ordinary umount failed, forcing... 2024-03-24T11:36:17.763 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:36:17.764 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:36:17.783 DEBUG:teuthology.orchestra.run.smithi100:> set -ex 2024-03-24T11:36:17.783 DEBUG:teuthology.orchestra.run.smithi100:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:36:17.841 DEBUG:teuthology.orchestra.run.smithi100:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2024-03-24T11:36:18.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:18.113+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14824 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:18.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:18.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:18.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:18.301+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:18.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:18.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:18.273+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:18.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:18.273+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:18.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:18.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:18.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:18.523+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:19.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:19.298+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:19.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:19.298+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:19.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:19.298+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:19.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:19.298+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:19.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:19.313+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:19.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:19.313+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:19.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:19.313+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:19.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:19.313+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:19.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:19.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:19.648 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:19.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:19.648 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:19.539+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:19.648 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:19.540+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:20.566 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:20.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:20.566 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:20.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:20.566 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:20.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:20.566 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:20.262+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:20.567 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:20.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:20.567 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:20.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:20.567 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:20.271+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:20.567 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:20.271+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:20.565+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:20.565+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:20.565+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:20.565+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:21.608 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:21.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:21.609 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:21.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:21.609 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:21.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:21.609 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:21.220+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:21.609 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:21.314+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:21.609 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:21.314+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:21.609 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:21.314+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:21.610 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:21.314+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:21.608+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:21.608+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:21.608+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:21.608+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:22.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:22.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:22.284+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:22.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:22.284+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:22.181+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:22.181+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:22.181+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:22.182+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:23.116 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:22.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:23.116 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:22.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:23.116 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:22.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:23.117 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:22.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:23.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:23.115+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14906 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:23.294+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:23.294+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:23.294+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:23.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:23.295+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:23.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:23.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:23.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:23.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:23.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:23.140+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:23.647+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:23.647+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:23.647+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:23.648+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:24.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:24.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:24.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:24.124+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:24.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:24.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:24.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:24.265+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:24.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:24.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:24.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:24.667+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:25.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:25.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:25.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:25.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:25.170+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:25.238+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:25.238+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:25.238+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:25.238+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:25.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:25.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:25.717+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:25.717+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:26.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:26.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:26.252+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:26.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:26.252+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:26.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:26.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:26.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:26.162+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:27.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:26.728+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:26.728+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:26.728+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:26.728+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:27.292+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:27.292+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:27.292+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:27.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:27.292+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:27.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:27.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:27.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:27.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:27.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:27.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:27.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:27.171+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:28.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:27.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:28.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:27.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:28.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:27.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:28.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:27.733+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:28.118+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 14989 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:28.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:28.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:28.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:28.197+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:28.261+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:28.261+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:28.261+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:28.261+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:28.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:28.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:28.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:28.744+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:29.231+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:29.231+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:29.231+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:29.231+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:29.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:29.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:29.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:29.208+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:29.700+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:29.700+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:29.700+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:29.700+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:30.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:30.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:30.198+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:30.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:30.198+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:30.213+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:30.213+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:30.213+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:30.213+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:30.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:30.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:30.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:30.725+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:31.174+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:31.174+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:31.174+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:31.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:31.174+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:31.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:31.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:31.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:31.193+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:31.968 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:31.762+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:31.969 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:31.762+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:31.969 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:31.762+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:31.969 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:31.763+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:32.170+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:32.170+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:32.170+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:32.170+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:32.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:32.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:32.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:32.151+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:33.122 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:32.772+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:33.122 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:32.772+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:33.122 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:32.772+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:33.122 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:32.773+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:33.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:33.120+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15071 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:33.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:33.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:33.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:33.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:33.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:33.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:33.210+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:33.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:33.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:33.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:33.196+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:33.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:33.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:33.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:33.808+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:34.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:34.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:34.175+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:34.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:34.175+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:34.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:34.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:34.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:34.204+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:35.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:34.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:35.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:34.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:35.127 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:34.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:35.127 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:34.805+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:35.125+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:35.125+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:35.125+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:35.126+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:35.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:35.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:35.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:35.223+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:35.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:35.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:35.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:35.784+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:36.577 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:36.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:36.577 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:36.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:36.577 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:36.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:36.577 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:36.145+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:36.577 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:36.215+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:36.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:36.215+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:36.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:36.215+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:36.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:36.215+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:36.800+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:36.800+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:36.800+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:36.800+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:37.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:37.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:37.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:37.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:37.158+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:37.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:37.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:37.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:37.220+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:38.112 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:37.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:38.113 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:37.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:38.113 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:37.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:38.113 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:37.838+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:38.148 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:38.121+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15154 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:38.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:38.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:38.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:38.250+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:38.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:38.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:38.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:38.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:38.208+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:38.793+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:38.793+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:38.793+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:38.793+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:39.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:39.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:39.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:39.224+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:39.159+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:39.159+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:39.159+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:39.159+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:39.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:39.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:39.817+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:39.817+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:40.155+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:40.155+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:40.155+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:40.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:40.156+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:40.177+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:40.177+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:40.177+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:40.177+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:40.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:40.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:40.808+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:40.808+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:41.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:41.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:41.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:41.169+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:41.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:41.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:41.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:41.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:41.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:41.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:41.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:41.143+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:41.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:41.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:41.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:41.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:42.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:42.195+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:42.195+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:42.195+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:42.196+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:42.161+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:42.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:42.161+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:42.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:42.161+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:42.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:42.161+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:43.125 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:42.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:43.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:42.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:43.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:42.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:43.126 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:42.815+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:43.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:43.124+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15235 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:43.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:43.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:43.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:43.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:43.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:43.198+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:43.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:43.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:43.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:43.200+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:43.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:43.200+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:43.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:43.200+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:43.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:43.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:43.779+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:43.780+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:44.240+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:44.240+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:44.240+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:44.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:44.241+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:44.165+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:44.165+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:44.165+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:44.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:44.165+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:44.781+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:44.781+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:44.781+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:44.782+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:45.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:45.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:45.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:45.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:45.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:45.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:45.647 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:45.206+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:45.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:45.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:45.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:45.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:45.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:45.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:45.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:45.166+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:45.767+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:45.767+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:45.767+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:45.768+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:46.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:46.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:46.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:46.250+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:46.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:46.154+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:46.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:46.154+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:46.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:46.154+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:46.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:46.155+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:46.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:46.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:46.804+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:46.804+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:47.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:47.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:47.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:47.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:47.276+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:47.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:47.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:47.172+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:47.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:47.172+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:48.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:47.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:48.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:47.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:48.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:47.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:48.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:47.814+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:48.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:48.127+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15319 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:48.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:48.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:48.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:48.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:48.247+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:48.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:48.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:48.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:48.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:48.143+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:48.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:48.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:48.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:49.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:48.791+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:49.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:49.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:49.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:49.294+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:49.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:49.187+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:49.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:49.187+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:49.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:49.187+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:49.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:49.187+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:49.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:49.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:49.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:49.764+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:50.311+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:50.311+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:50.311+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:50.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:50.312+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:50.204+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:50.204+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:50.204+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:50.204+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:50.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:50.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:50.764+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:51.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:50.764+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:51.315+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:51.315+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:51.315+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:51.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:51.316+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:51.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:51.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:51.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:51.193+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:51.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:51.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:51.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:51.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:52.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:52.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:52.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:52.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:52.273+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:52.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:52.217+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:52.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:52.217+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:52.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:52.217+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:52.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:52.218+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:53.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:52.836+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:53.131 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:52.836+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:53.131 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:52.836+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:53.131 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:52.837+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:53.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:53.129+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15399 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:53.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:53.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:53.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:53.308+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:53.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:53.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:53.268+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:53.268+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:53.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:53.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:53.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:53.838+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:54.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:54.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:54.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:54.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:54.309+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:54.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:54.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:54.303+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:54.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:54.304+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:54.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:54.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:54.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:54.815+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:55.353+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:55.353+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:55.353+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:55.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:55.354+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:55.309+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:55.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:55.309+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:55.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:55.309+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:55.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:55.309+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:55.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:55.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:55.837+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:55.837+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:56.544 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:56.259+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:56.544 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:56.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:56.544 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:56.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:56.545 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:56.260+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:56.545 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:56.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:56.545 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:56.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:56.545 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:56.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:56.545 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:56.361+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:56.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:56.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:56.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:56.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:57.333+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:57.333+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:57.333+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:57.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:57.333+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:57.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:57.257+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:57.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:57.257+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:57.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:57.257+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:57.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:57.257+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:58.133 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:57.803+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:58.133 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:57.803+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:58.133 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:57.803+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:58.133 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:57.803+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:36:58.132+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15482 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:36:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:58.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:58.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:58.264+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:58.265+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:58.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:58.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:58.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:58.344+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:58.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:58.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:58.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:36:59.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:58.812+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:36:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:59.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:59.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:59.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:36:59.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:36:59.314+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:36:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:59.290+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:59.290+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:59.290+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:36:59.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:36:59.290+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:59.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:59.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:59.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:36:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:36:59.776+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:00.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:00.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:00.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:00.288+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:00.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:00.323+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:00.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:00.323+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:00.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:00.323+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:00.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:00.323+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:00.799+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:00.799+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:00.799+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:00.800+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:01.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:01.284+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:01.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:01.284+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:01.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:01.284+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:01.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:01.284+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:01.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:01.283+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:01.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:01.283+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:01.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:01.283+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:01.645 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:01.283+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:01.794+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:01.794+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:01.794+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:01.794+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:02.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:02.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:02.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:02.294+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:02.239+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:02.239+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:02.239+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:02.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:02.239+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:03.134 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:02.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:03.134 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:02.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:03.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:02.777+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:03.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:02.777+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:03.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:03.134+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15563 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:03.218+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:03.218+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:03.219+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:03.219+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:03.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:03.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:03.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:03.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:03.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:03.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:03.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:03.259+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:03.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:03.767+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:03.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:03.767+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:03.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:03.767+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:03.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:03.767+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:04.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:04.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:04.234+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:04.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:04.235+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:04.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:04.191+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:04.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:04.191+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:04.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:04.191+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:04.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:04.192+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:04.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:04.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:04.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:04.812+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:05.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:05.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:05.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:05.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:05.249+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:05.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:05.209+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:05.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:05.209+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:05.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:05.209+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:05.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:05.209+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:05.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:05.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:05.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:05.829+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:06.258+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:06.258+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:06.258+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:06.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:06.258+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:06.220+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:06.220+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:06.220+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:06.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:06.220+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:06.785+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:06.785+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:06.785+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:06.786+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:07.265+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:07.265+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:07.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:07.265+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:07.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:07.265+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:07.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:07.212+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:07.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:07.212+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:07.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:07.212+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:07.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:07.212+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:08.138 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:07.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:08.138 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:07.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:08.138 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:07.819+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:08.139 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:07.820+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:08.137+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15647 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:08.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:08.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:08.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:08.303+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:08.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:08.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:08.216+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:08.216+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:08.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:08.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:08.790+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:09.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:08.790+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:09.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:09.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:09.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:09.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:09.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:09.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:09.275+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:09.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:09.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:09.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:09.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:09.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:09.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:09.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:09.197+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:09.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:09.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:09.798+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:09.798+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:10.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:10.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:10.259+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:10.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:10.260+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:10.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:10.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:10.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:10.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:10.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:10.232+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:10.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:10.232+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:10.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:10.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:10.847+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:10.847+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:11.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:11.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:11.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:11.273+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:11.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:11.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:11.208+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:11.208+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:11.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:11.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:11.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:11.830+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:12.267+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:12.267+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:12.267+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:12.267+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:12.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:12.205+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:12.205+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:12.205+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:12.205+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:13.140 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:12.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:13.141 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:12.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:13.141 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:12.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:13.141 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:12.842+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:13.139+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15727 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:13.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:13.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:13.192+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:13.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:13.192+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:13.264+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:13.264+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:13.264+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:13.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:13.264+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:13.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:13.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:13.809+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:13.809+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:14.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:14.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:14.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:14.182+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:14.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:14.297+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:14.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:14.297+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:14.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:14.297+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:14.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:14.297+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:14.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:14.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:14.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:14.839+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:15.271+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:15.271+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:15.271+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:15.272+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:15.203+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:15.203+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:15.203+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:15.204+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:15.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:15.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:15.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:15.798+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:16.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:16.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:16.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:16.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:16.248+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:16.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:16.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:16.156+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:16.156+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:16.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:16.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:16.838+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:16.839+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:17.282+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:17.282+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:17.282+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:17.282+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:17.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:17.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:17.199+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:17.199+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:17.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:17.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:17.801+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:17.801+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:18.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:18.141+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15810 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:18.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:18.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:18.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:18.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:18.197+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:18.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:18.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:18.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:18.315+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:18.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:18.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:18.775+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:18.775+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:19.365+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:19.365+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:19.365+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:19.365+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:19.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:19.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:19.210+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:19.210+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:19.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:19.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:19.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:19.741+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:20.374+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:20.374+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:20.374+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:20.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:20.374+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:20.207+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:20.207+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:20.207+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:20.207+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:21.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:20.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:21.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:20.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:21.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:20.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:21.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:20.783+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:21.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:21.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:21.182+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:21.182+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:21.701 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:21.395+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:21.701 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:21.395+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:21.701 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:21.395+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:21.701 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:21.395+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:21.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:21.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:21.778+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:21.778+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:22.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:22.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:22.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:22.352+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:22.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:22.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:22.166+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:22.166+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:22.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:22.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:22.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:22.760+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:23.588 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:23.144+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15891 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:23.588 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:23.335+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:23.588 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:23.335+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:23.588 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:23.335+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:23.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:23.336+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:23.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:23.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:23.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:23.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:23.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:23.158+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:23.589 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:23.158+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:23.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:23.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:23.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:23.808+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:24.376 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:24.112+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:24.376 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:24.112+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:24.376 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:24.112+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:24.376 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:24.113+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:24.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:24.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:24.375+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:24.375+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:25.134 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:24.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:25.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:24.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:25.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:24.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:25.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:24.843+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:25.133+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:25.133+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:25.133+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:25.134+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:25.363+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:25.363+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:25.363+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:25.363+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:26.124+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:26.124+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:26.124+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:26.124+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:25.881+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:25.881+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:25.881+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:25.882+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:26.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:26.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:26.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:26.344+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:27.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:26.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:27.120 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:26.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:27.121 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:26.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:27.121 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:26.841+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:27.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:27.120+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:27.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:27.120+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:27.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:27.120+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:27.377 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:27.120+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:27.376+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:27.376+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:27.376+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:27.376+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:28.140 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:27.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:28.141 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:27.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:28.141 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:27.861+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:28.141 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:27.862+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:28.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:28.146+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 15974 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:28.139+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:28.139+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:28.139+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:28.140+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:28.879 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:28.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:28.879 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:28.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:28.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:28.414+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:28.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:28.414+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:28.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:28.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:29.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:28.878+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:29.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:28.879+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:29.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:29.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:29.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:29.116+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:29.871 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:29.450+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:29.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:29.450+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:29.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:29.450+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:29.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:29.450+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:29.870+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:29.870+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:29.870+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:29.871+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:30.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:30.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:30.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:30.116+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:30.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:30.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:30.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:30.414+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:30.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:30.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:30.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:30.911+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:31.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:31.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:31.080+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:31.080+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:31.702 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:31.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:31.702 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:31.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:31.702 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:31.397+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:31.702 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:31.398+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:32.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:31.894+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:32.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:31.894+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:32.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:31.894+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:32.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:31.895+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:32.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:32.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:32.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:32.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:32.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:32.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:32.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:32.044+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:32.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:32.384+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:32.384+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:32.384+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:32.384+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:33.150 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:33.028+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:33.150 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:33.028+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:33.150 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:33.028+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:33.151 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:33.028+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:33.151 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:32.892+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:33.151 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:32.892+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:33.151 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:32.892+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:33.151 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:32.893+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:33.408 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:33.149+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16054 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:33.408 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:33.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:33.408 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:33.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:33.408 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:33.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:33.408 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:33.361+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:33.890+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:33.890+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:33.890+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:33.890+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:34.066+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:34.066+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:34.066+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:34.066+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:34.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:34.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:34.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:34.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:34.409+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:34.936+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:34.936+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:34.936+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:34.936+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:35.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:35.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:35.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:35.090+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:35.402+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:35.402+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:35.402+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:35.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:35.402+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:35.923+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:35.923+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:35.923+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:35.923+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:36.042+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:36.042+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:36.042+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:36.042+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:36.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:36.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:36.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:36.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:36.449+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:37.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:37.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:37.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:37.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:37.035+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:36.955+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:37.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:36.955+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:37.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:36.955+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:37.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:36.955+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:37.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:37.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:37.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:37.439+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:38.276 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:38.151+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16137 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:38.276 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:38.052+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:38.277 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:38.052+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:38.277 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:38.052+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:38.277 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:38.053+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:38.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:37.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:38.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:37.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:38.277 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:37.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:38.278 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:37.985+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:38.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:38.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:38.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:38.413+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:39.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:39.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:39.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:39.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:39.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:39.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:39.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:39.035+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:39.381 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:38.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:39.381 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:38.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:39.381 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:38.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:39.381 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:38.982+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:39.379+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:39.379+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:39.379+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:39.380+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:40.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:40.075+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:40.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:40.075+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:40.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:40.075+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:40.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:40.075+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:40.371 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:39.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:40.371 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:39.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:40.371 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:39.994+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:40.371 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:39.994+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:40.369+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:40.369+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:40.369+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:40.369+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:41.389 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:41.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:41.389 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:41.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:41.390 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:41.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:41.390 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:41.026+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:41.390 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.031+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:41.390 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.031+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:41.390 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.031+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:41.390 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.031+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:41.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:41.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:41.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:41.389+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:42.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.988+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:41.988+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:42.010+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:42.010+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:42.010+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:42.010+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:42.392+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:42.392+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:42.392+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:42.392+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:43.279 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:43.154+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16217 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:43.279 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:42.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:42.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:42.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:42.944+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:43.042+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:43.042+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:43.042+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:43.280 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:43.043+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:43.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:43.437+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:43.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:43.437+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:43.589 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:43.437+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:43.590 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:43.437+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:44.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:44.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:44.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:44.093+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:43.934+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:43.934+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:43.934+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:43.934+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:44.452+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:44.452+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:44.452+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:44.453+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:45.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:45.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:45.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:45.046+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:44.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:44.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:44.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:45.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:44.913+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:45.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:45.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:45.472+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:45.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:45.472+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:46.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:46.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:46.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:46.064+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:45.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:45.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:45.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:46.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:45.929+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:46.886 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:46.486+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:46.886 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:46.486+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:46.887 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:46.486+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:46.887 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:46.486+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:47.049+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:47.049+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:47.049+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:47.049+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:46.885+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:46.885+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:46.885+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:46.886+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:47.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:47.496+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:47.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:47.496+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:47.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:47.496+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:47.877 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:47.496+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:47.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:47.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:47.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:47.877+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:48.088+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:48.088+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:48.088+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:48.088+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:48.525 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:48.156+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16300 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:48.829 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:48.525+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:48.830 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:48.525+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:48.830 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:48.525+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:48.830 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:48.525+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:49.078+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:49.078+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:49.078+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:49.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:49.079+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:48.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:48.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:48.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:48.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:49.792 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:49.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:49.792 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:49.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:49.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:49.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:49.793 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:49.529+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:49.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:49.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:49.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:49.792+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:50.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:50.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:50.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:50.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:50.039+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:50.784 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:50.517+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:50.784 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:50.517+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:50.785 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:50.517+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:50.785 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:50.517+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:50.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:50.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:50.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:50.784+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:51.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:51.022+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:51.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:51.022+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:51.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:51.022+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:51.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:51.023+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:51.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:51.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:51.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:51.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:51.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:51.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:51.840 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:51.746+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:51.841 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:51.561+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:51.841 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:51.561+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:51.841 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:51.561+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:51.841 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:51.561+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:52.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:52.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:52.039+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:52.040+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:52.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:52.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:52.754+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:52.754+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:52.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:52.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:52.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:52.526+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:53.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:53.159+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16380 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:53.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:53.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:53.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:53.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:53.057+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:53.751 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:53.488+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:53.751 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:53.488+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:53.751 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:53.488+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:53.751 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:53.488+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:54.025 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:53.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:54.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:53.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:54.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:53.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:54.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:53.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:54.025+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:54.025+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:54.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:54.025+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:54.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:54.025+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:54.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:54.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:54.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:54.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:54.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:54.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:54.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:54.463+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:55.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:54.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:55.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:54.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:55.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:54.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:55.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:54.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:55.069+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:55.069+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:55.069+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:55.070+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:55.725 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:55.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:55.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:55.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:55.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:55.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:55.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:55.428+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:56.110 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:55.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:56.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:55.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:56.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:55.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:56.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:55.725+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:56.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:56.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:56.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:56.110+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:56.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:56.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:56.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:56.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:56.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:56.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:56.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:56.414+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:57.123 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:56.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:57.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:56.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:57.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:56.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:57.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:56.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:57.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:57.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:57.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:57.122+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:57.754 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:57.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:57.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:57.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:57.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:57.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:57.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:57.435+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:58.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:57.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:58.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:57.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:58.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:57.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:58.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:57.753+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:37:58.160+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16463 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:37:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:58.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:58.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:58.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:58.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:58.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:58.128+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:58.754 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:58.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:58.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:58.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:58.755 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:58.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:58.756 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:58.426+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:37:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:58.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:58.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:58.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:37:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:58.753+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:59.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:59.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:59.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:37:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:37:59.175+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:37:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:59.417+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:59.417+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:59.417+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:37:59.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:37:59.417+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:59.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:59.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:59.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:37:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:37:59.786+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:00.435 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:00.156+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:00.436 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:00.156+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:00.436 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:00.156+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:00.436 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:00.156+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:00.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:00.434+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:00.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:00.434+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:00.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:00.434+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:00.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:00.434+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:00.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:00.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:00.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:00.806+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:01.419 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:01.142+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:01.419 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:01.142+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:01.420 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:01.142+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:01.420 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:01.142+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:01.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:01.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:01.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:01.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:01.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:01.418+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:01.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:01.418+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:01.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:01.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:01.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:01.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:02.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:02.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:02.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:02.407+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:02.158+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:02.158+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:02.158+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:02.159+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:02.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:02.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:02.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:02.777+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:03.439 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:03.164+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16544 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:03.440 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:03.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:03.440 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:03.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:03.440 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:03.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:03.440 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:03.194+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:03.756 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:03.438+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:03.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:03.438+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:03.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:03.438+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:03.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:03.439+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:04.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:03.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:04.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:03.756+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:04.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:03.756+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:04.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:03.756+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:04.159+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:04.159+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:04.159+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:04.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:04.159+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:04.715 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:04.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:04.715 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:04.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:04.716 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:04.446+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:04.716 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:04.446+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:05.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:04.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:05.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:04.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:05.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:04.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:05.133 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:04.715+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:05.131+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:05.131+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:05.131+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:05.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:05.132+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:05.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:05.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:05.448+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:05.449+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:05.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:05.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:05.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:05.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:05.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:05.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:05.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:05.689+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:06.118+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:06.118+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:06.118+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:06.118+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:06.494+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:06.494+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:06.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:06.494+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:06.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:06.494+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:06.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:06.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:06.679+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:06.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:06.680+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:07.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:07.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:07.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:07.082+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:07.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:07.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:07.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:07.475+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:07.669+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:07.669+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:07.669+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:07.669+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:08.167+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16628 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:08.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:08.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:08.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:08.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:08.065+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:08.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:08.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:08.492+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:08.493+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:08.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:08.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:08.715+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:08.715+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:09.047+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:09.047+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:09.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:09.047+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:09.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:09.047+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:09.711 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:09.457+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:09.712 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:09.457+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:09.712 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:09.457+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:09.712 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:09.458+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:10.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:09.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:10.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:09.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:10.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:09.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:10.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:09.711+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:10.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:10.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:10.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:10.064+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:10.747 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:10.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:10.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:10.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:10.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:10.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:10.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:10.424+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:11.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:10.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:11.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:10.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:11.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:10.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:11.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:10.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:11.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:11.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:11.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:11.096+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:11.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:11.444+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:11.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:11.444+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:11.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:11.444+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:11.705 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:11.444+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:12.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:11.726+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:12.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:11.726+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:12.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:11.726+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:12.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:11.726+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:12.138+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:12.138+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:12.138+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:12.138+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:12.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:12.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:12.762 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:12.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:12.763 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:12.478+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:12.763 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:12.479+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:13.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:12.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:13.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:12.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:13.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:12.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:13.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:12.762+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:13.523 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:13.169+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16709 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:13.523 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:13.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:13.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:13.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:13.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:13.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:13.524 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:13.149+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:13.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:13.522+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:13.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:13.522+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:13.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:13.522+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:13.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:13.523+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:14.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:13.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:14.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:13.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:14.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:13.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:14.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:13.808+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:14.135+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:14.135+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:14.135+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:14.136+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:14.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:14.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:14.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:14.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:14.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:14.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:14.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:14.487+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:14.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:14.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:14.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:14.770+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:15.463 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:15.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:15.463 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:15.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:15.464 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:15.172+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:15.464 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:15.172+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:15.732 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:15.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:15.732 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:15.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:15.732 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:15.462+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:15.732 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:15.463+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:16.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:15.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:16.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:15.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:16.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:15.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:16.138 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:15.731+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:16.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:16.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:16.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:16.137+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:16.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:16.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:16.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:16.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:16.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:16.485+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:16.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:16.485+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:16.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:16.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:16.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:16.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:17.440 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:17.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:17.441 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:17.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:17.441 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:17.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:17.441 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:17.183+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:17.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:17.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:17.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:17.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:17.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:17.439+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:17.773 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:17.440+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:17.919 DEBUG:teuthology.orchestra.run:got remote process result: 124 2024-03-24T11:38:17.920 DEBUG:tasks.cephfs.mount:Force/lazy unmounting on client.0 2024-03-24T11:38:17.920 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:38:17.920 DEBUG:teuthology.orchestra.run.smithi100:> sudo umount --lazy --force /home/ubuntu/cephtest/mnt.0 2024-03-24T11:38:17.960 DEBUG:teuthology.orchestra.run:got remote process result: 2 2024-03-24T11:38:17.960 INFO:tasks.workunit.client.0.smithi100.stderr:make: *** INTERNAL: readdir: Input/output error. Stop. 2024-03-24T11:38:17.961 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.0... 2024-03-24T11:38:17.961 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.client.0 /home/ubuntu/cephtest/clone.client.0 2024-03-24T11:38:17.970 INFO:tasks.cephfs.mount:Cleaning up mount ubuntu@smithi100.front.sepia.ceph.com 2024-03-24T11:38:17.970 INFO:teuthology.orchestra.run:Running command with timeout 300 2024-03-24T11:38:17.970 DEBUG:teuthology.orchestra.run.smithi100:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2024-03-24T11:38:18.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:17.771+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:18.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:17.771+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:18.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:17.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:18.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:17.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:18.036 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-03-24T11:38:18.036 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/proc/self/mounts of=/dev/stdout 2024-03-24T11:38:18.296 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:18.171+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16793 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:18.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:18.231+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:18.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:18.231+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:18.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:18.231+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:18.297 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:18.231+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:18.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:18.419+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:18.419+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:18.419+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:18.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:18.419+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:18.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:18.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:18.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:18.758+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:19.244+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:19.244+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:19.244+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:19.244+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:19.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:19.370+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:19.370+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:19.370+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:19.371+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:19.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:19.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:19.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:19.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:20.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:20.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:20.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:20.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:20.381+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:20.264+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:20.264+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:20.264+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:20.265+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:20.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:20.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:20.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:20.719+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:21.300+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:21.300+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:21.300+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:21.300+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:21.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:21.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:21.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:21.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:21.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:21.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:21.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:21.426+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:21.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:21.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:21.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:21.690+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:22.560 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:22.286+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:22.560 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:22.286+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:22.560 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:22.286+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:22.560 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:22.287+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:22.561 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:22.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:22.561 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:22.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:22.561 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:22.409+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:22.561 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:22.410+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:22.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:22.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:22.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:22.676+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:23.593 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:23.174+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16874 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:23.593 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:23.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:23.593 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:23.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:23.594 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:23.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:23.594 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:23.287+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:23.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:23.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:23.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:23.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:23.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:23.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:23.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:23.401+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:23.631+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:23.631+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:23.631+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:23.631+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:24.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:24.440+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:24.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:24.440+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:24.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:24.440+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:24.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:24.441+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:24.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:24.315+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:24.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:24.315+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:24.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:24.315+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:24.587 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:24.316+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:24.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:24.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:24.585+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:24.585+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:25.604 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:25.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:25.604 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:25.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:25.604 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:25.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:25.605 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:25.304+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:25.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:25.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:25.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:25.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:25.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:25.404+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:25.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:25.404+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:25.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:25.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:25.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:25.602+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:26.296+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:26.296+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:26.296+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:26.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:26.296+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:26.365+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:26.365+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:26.365+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:26.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:26.365+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:26.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:26.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:26.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:26.646+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:27.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:27.329+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:27.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:27.329+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:27.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:27.329+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:27.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:27.329+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:27.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:27.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:27.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:27.391+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:27.641+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:27.641+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:27.641+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:27.641+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:28.637 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:28.177+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 16958 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:28.638 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:28.374+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:28.638 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:28.374+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:28.638 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:28.374+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:28.638 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:28.375+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:28.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:28.373+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:28.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:28.373+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:28.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:28.373+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:28.640 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:28.373+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:28.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:28.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:28.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:28.636+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:29.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:29.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:29.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:29.421+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:29.330+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:29.330+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:29.330+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:29.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:29.331+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:29.675+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:29.675+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:29.675+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:29.675+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:30.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:30.394+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:30.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:30.394+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:30.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:30.394+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:30.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:30.394+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:30.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:30.357+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:30.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:30.357+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:30.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:30.357+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:30.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:30.358+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:30.627+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:30.627+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:30.627+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:30.627+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:31.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:31.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:31.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:31.441+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:31.349+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:31.349+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:31.349+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:31.349+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:31.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:31.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:31.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:31.643+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:32.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:32.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:32.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:32.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:32.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:32.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:32.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:32.480+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:32.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:32.307+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:32.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:32.307+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:32.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:32.307+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:32.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:32.307+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:32.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:32.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:32.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:32.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:32.612+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:33.503 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:33.179+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17039 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:33.503 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:33.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:33.504 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:33.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:33.504 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:33.350+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:33.504 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:33.350+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:33.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:33.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:33.502+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:33.503+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:33.607+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:33.607+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:33.607+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:33.607+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:34.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:34.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:34.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:34.481+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:34.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:34.594+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:34.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:34.594+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:34.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:34.594+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:34.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:34.594+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:34.358+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:34.358+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:34.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:34.358+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:34.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:34.358+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:35.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:35.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:35.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:35.509+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:35.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:35.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:35.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:35.577+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:35.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:35.394+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:35.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:35.394+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:35.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:35.394+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:35.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:35.394+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:36.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:36.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:36.542+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:36.542+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:36.567+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:36.567+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:36.567+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:36.567+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:36.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:36.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:36.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:36.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:36.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:36.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:36.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:36.352+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:37.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:37.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:37.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:37.533+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:37.543+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:37.543+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:37.543+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:37.543+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:37.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:37.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:37.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:37.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:37.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:37.361+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:37.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:37.361+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:38.556 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:38.181+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17122 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:38.557 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:38.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:38.557 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:38.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:38.557 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:38.314+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:38.558 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:38.314+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:38.555+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:38.555+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:38.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:38.555+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:38.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:38.555+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:38.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:38.559+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:38.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:38.559+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:38.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:38.559+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:38.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:38.559+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:39.341+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:39.341+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:39.341+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:39.341+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:39.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:39.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:39.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:39.584+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:39.545+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:39.545+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:39.545+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:39.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:39.545+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:40.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:40.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:40.578 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:40.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:40.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:40.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:40.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:40.303+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:40.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:40.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:40.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:40.626+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:40.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:40.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:40.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:40.577+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:41.597 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:41.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:41.597 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:41.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:41.597 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:41.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:41.597 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:41.301+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:41.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:41.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:41.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:41.650+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:41.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:41.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:41.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:41.596+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:42.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:42.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:42.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:42.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:42.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:42.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:42.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:42.348+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:42.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:42.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:42.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:42.639+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:42.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:42.625+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:42.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:42.625+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:42.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:42.625+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:42.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:42.626+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:43.580 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:43.185+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17203 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:43.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:43.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:43.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:43.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:43.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:43.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:43.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:43.340+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:43.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:43.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:43.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:43.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:43.579+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:43.579+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:43.579+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:43.580+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:44.305+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:44.305+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:44.305+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:44.305+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:44.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:44.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:44.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:44.674+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:44.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:44.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:44.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:44.577+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:45.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:45.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:45.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:45.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:45.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:45.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:45.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:45.347+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:45.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:45.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:45.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:45.682+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:45.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:45.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:45.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:45.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:45.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:45.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:45.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:45.618+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:46.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:46.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:46.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:46.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:46.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:46.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:46.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:46.300+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:46.706+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:46.706+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:46.706+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:46.707+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:46.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:46.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:46.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:46.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:46.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:46.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:46.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:46.615+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:47.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:47.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:47.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:47.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:47.603 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:47.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:47.603 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:47.287+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:47.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:47.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:47.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:47.660+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:47.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:47.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:47.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:47.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:47.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:47.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:47.602+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:48.623 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:48.187+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17288 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:48.623 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:48.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:48.623 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:48.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:48.624 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:48.308+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:48.624 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:48.309+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:48.670+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:48.670+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:48.670+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:48.671+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:48.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:48.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:48.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:48.622+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:49.613 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:49.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:49.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:49.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:49.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:49.270+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:49.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:49.270+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:49.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:49.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:49.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:49.640+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:49.613+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:49.613+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:49.613+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:49.613+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:50.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:50.266+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:50.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:50.266+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:50.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:50.266+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:50.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:50.266+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:50.683+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:50.683+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:50.683+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:50.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:50.684+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:50.617+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:50.617+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:50.617+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:50.617+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:51.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:51.230+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:51.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:51.230+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:51.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:51.230+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:51.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:51.231+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:51.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:51.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:51.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:51.695+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:51.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:51.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:51.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:51.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:51.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:51.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:51.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:51.595+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:52.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:52.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:52.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:52.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:52.607 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:52.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:52.607 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:52.279+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:52.665+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:52.665+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:52.665+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:52.666+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:52.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:52.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:52.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:52.606+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:53.594 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:53.190+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17368 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:53.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:53.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:53.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:53.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:53.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:53.256+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:53.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:53.256+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:53.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:53.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:53.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:53.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:53.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:53.629+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:53.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:53.629+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:53.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:53.629+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:53.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:53.630+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:54.591 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:54.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:54.591 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:54.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:54.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:54.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:54.592 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:54.224+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:54.590+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:54.590+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:54.590+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:54.591+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:54.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:54.683+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:54.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:54.683+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:54.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:54.683+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:54.898 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:54.683+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:55.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:55.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:55.607 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:55.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:55.607 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:55.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:55.607 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:55.223+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:55.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:55.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:55.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:55.606+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:55.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:55.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:55.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:55.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:55.667+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:55.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:55.667+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:56.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:56.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:56.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:56.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:56.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:56.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:56.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:56.188+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:56.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:56.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:56.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:56.573+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:56.661+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:56.661+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:56.661+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:56.661+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:57.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:57.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:57.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:57.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:57.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:57.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:57.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:57.197+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:57.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:57.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:57.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:57.651+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:57.536+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:57.536+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:57.536+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:57.536+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:58.515 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:38:58.193+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17452 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:38:58.516 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:58.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:58.516 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:58.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:58.516 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:58.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:58.517 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:58.170+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:58.682+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:58.682+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:58.682+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:58.682+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:58.514+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:58.514+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:58.514+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:58.514+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:59.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:59.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:59.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:38:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:38:59.206+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:38:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:59.478+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:59.478+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:59.478+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:38:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:38:59.478+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:38:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:59.720+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:59.720+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:59.720+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:38:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:38:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:38:59.721+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:00.486 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:00.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:00.486 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:00.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:00.486 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:00.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:00.486 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:00.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:00.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:00.727+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:00.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:00.727+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:00.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:00.727+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:00.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:00.727+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:00.751 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:00.485+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:00.751 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:00.485+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:00.751 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:00.485+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:00.751 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:00.485+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:01.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:01.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:01.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:01.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:01.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:01.153+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:01.710 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:01.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:01.710 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:01.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:01.710 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:01.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:01.710 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:01.505+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:02.104 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:01.745+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:02.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:01.745+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:02.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:01.745+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:02.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:01.746+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:02.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:02.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:02.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:02.104+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:02.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:02.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:02.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:02.743+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:02.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:02.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:02.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:02.512+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:03.318 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:03.195+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17533 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:03.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:03.061+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:03.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:03.061+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:03.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:03.061+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:03.319 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:03.061+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:03.595 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:03.470+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:03.595 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:03.470+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:03.596 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:03.470+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:03.596 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:03.470+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:03.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:03.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:03.780+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:03.780+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:04.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:04.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:04.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:04.109+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:04.766 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:04.475+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:04.766 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:04.475+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:04.767 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:04.475+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:04.767 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:04.475+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:05.137 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:04.765+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:05.137 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:04.765+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:05.138 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:04.765+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:05.138 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:04.766+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:05.136+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:05.136+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:05.136+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:05.136+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:05.732 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:05.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:05.732 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:05.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:05.732 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:05.442+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:05.733 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:05.442+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:06.139 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:05.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:06.140 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:05.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:06.140 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:05.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:06.140 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:05.732+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:06.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:06.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:06.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:06.139+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:06.742 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:06.468+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:06.743 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:06.468+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:06.743 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:06.468+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:06.743 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:06.468+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:07.101 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:06.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:07.102 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:06.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:07.102 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:06.742+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:07.102 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:06.742+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:07.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:07.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:07.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:07.101+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:07.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:07.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:07.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:07.698+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:07.487+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:07.487+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:07.487+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:07.487+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:08.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:08.198+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17617 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:08.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:08.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:08.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:08.087+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:08.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:08.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:08.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:08.729+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:08.490+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:08.490+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:08.490+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:08.490+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:09.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:09.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:09.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:09.046+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:09.702+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:09.702+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:09.702+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:09.702+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:09.465+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:09.465+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:09.465+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:09.465+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:10.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:10.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:10.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:10.067+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:10.436+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:10.436+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:10.436+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:10.436+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:10.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:10.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:10.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:10.669+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:11.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:11.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:11.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:11.112+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:11.463+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:11.463+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:11.463+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:11.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:11.463+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:11.703+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:11.703+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:11.703+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:11.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:11.703+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:12.159 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:12.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:12.160 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:12.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:12.160 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:12.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:12.160 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:12.091+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:12.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:12.662+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:12.826 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:12.662+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:12.827 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:12.662+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:12.827 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:12.662+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:12.827 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:12.492+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:12.827 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:12.492+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:12.827 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:12.492+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:12.827 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:12.492+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:13.200+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17698 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:13.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:13.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:13.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:13.116+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:13.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:13.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:13.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:13.617+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:13.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:13.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:13.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:13.512+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:14.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:14.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:14.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:14.081+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:14.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:14.640+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:14.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:14.640+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:14.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:14.640+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:14.784 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:14.640+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:14.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:14.503+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:14.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:14.503+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:14.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:14.503+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:14.785 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:14.504+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:15.032+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:15.032+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:15.032+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:15.032+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:15.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:15.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:15.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:15.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:15.538+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:15.538+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:15.538+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:15.539+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:16.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:16.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:16.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:16.005+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:16.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:16.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:16.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:16.684+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:16.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:16.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:16.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:16.553+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:17.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:17.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:17.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:17.052+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:17.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:17.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:17.639+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:17.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:17.639+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:17.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:17.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:17.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:17.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:17.595+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:18.327 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:18.201+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17782 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:18.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:18.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:18.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:18.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:18.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:18.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:18.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:18.046+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:18.624+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:18.624+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:18.624+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:18.624+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:18.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:18.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:18.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:18.674+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:19.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:19.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:19.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:19.048+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:19.670+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:19.670+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:19.670+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:19.670+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:19.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:19.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:19.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:19.636+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:20.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:20.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:20.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:20.084+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:20.928 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:20.654+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:20.928 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:20.654+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:20.928 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:20.654+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:20.928 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:20.654+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:20.929 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:20.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:20.929 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:20.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:20.929 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:20.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:20.929 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:20.686+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:21.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:21.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:21.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:21.088+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:21.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:21.088+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:22.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:21.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:22.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:21.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:22.135 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:21.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:22.136 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:21.655+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:22.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:21.655+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:22.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:21.655+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:22.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:21.655+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:22.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:21.655+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:22.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:22.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:22.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:22.135+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:22.689+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:22.689+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:22.689+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:22.690+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:22.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:22.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:22.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:22.695+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:23.408 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:23.205+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17863 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:23.409 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:23.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:23.409 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:23.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:23.409 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:23.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:23.409 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:23.154+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:23.714 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:23.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:23.714 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:23.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:23.715 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:23.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:23.715 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:23.652+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:23.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:23.713+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:23.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:23.713+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:23.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:23.713+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:23.997 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:23.714+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:24.195+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:24.195+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:24.195+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:24.196+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:24.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:24.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:24.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:24.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:24.669+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:24.632+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:24.632+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:24.632+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:24.632+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:25.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:25.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:25.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:25.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:25.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:25.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:25.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:25.223+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:25.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:25.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:25.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:25.660+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:25.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:25.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:25.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:25.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:25.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:25.619+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:26.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:26.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:26.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:26.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:26.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:26.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:26.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:26.226+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:26.647+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:26.647+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:26.647+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:26.647+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:26.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:26.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:26.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:26.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:26.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:26.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:26.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:26.638+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:27.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:27.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:27.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:27.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:27.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:27.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:27.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:27.273+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:27.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:27.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:27.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:27.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:27.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:27.680+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:27.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:27.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:27.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:27.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:27.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:27.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:27.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:27.593+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:28.584 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:28.208+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 17947 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:28.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:28.277+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:28.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:28.277+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:28.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:28.277+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:28.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:28.278+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:28.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:28.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:28.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:28.584+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:28.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:28.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:28.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:28.714+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:29.595 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:29.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:29.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:29.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:29.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:29.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:29.596 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:29.272+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:29.851 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:29.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:29.851 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:29.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:29.851 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:29.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:29.851 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:29.752+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:29.851 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:29.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:29.852 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:29.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:29.852 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:29.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:29.852 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:29.623+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:30.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:30.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:30.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:30.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:30.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:30.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:30.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:30.275+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:30.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:30.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:30.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:30.619+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:30.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:30.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:30.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:30.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:30.773+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:31.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:31.251+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:31.611 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:31.251+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:31.611 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:31.251+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:31.611 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:31.251+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:31.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:31.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:31.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:31.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:31.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:31.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:31.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:31.610+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:32.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:32.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:32.247+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:32.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:32.247+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:32.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:32.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:32.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:32.790+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:32.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:32.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:32.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:32.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:33.597 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:33.211+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18028 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:33.598 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:33.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:33.598 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:33.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:33.598 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:33.293+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:33.598 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:33.294+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:33.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:33.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:33.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:33.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:33.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:33.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:33.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:33.812+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:33.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:33.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:33.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:33.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:33.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:33.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:33.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:33.687+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:34.329+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:34.329+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:34.329+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:34.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:34.329+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:34.825+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:34.825+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:34.825+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:34.825+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:34.703+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:34.703+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:34.703+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:34.704+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:35.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:35.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:35.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:35.295+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:35.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:35.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:35.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:35.732+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:35.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:35.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:35.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:35.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:36.324+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:36.324+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:36.324+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:36.325+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:36.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:36.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:36.718+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:36.718+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:36.867+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:36.867+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:36.867+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:36.867+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:37.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:37.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:37.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:37.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:37.340+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:37.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:37.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:37.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:37.866+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:38.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:37.714+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:38.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:37.714+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:38.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:37.714+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:38.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:37.714+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:38.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:38.213+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18112 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:38.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:38.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:38.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:38.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:38.348+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:38.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:38.349+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:38.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:38.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:38.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:39.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:38.761+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:38.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:38.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:39.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:38.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:39.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:38.819+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:39.372+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:39.372+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:39.372+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:39.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:39.373+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:39.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:39.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:39.760+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:40.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:39.760+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:39.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:39.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:39.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:39.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:40.357+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:40.357+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:40.357+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:40.357+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:40.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:40.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:40.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:40.758+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:40.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:40.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:40.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:40.810+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:41.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:41.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:41.391+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:41.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:41.391+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:41.739+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:41.739+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:41.739+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:42.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:41.740+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:41.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:41.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:41.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:41.808+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:42.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:42.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:42.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:42.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:42.386+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:42.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:42.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:42.735+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:42.735+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:42.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:42.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:42.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:42.784+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:43.598 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:43.216+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18193 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:43.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:43.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:43.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:43.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:43.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:43.386+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:43.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:43.386+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:43.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:43.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:43.783+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:43.784+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:43.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:43.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:43.792+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:43.792+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:44.753 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:44.400+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:44.753 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:44.400+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:44.753 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:44.400+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:44.753 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:44.401+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:44.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:44.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:44.752+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:45.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:44.752+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:44.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:44.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:44.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:44.766+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:45.364+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:45.364+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:45.364+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:45.364+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:45.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:45.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:45.797+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:45.797+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:45.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:45.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:45.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:45.721+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:46.385+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:46.385+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:46.385+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:46.385+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:46.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:46.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:46.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:47.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:46.843+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:46.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:46.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:46.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:46.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:47.797 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:47.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:47.797 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:47.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:47.797 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:47.666+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:47.797 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:47.666+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:47.797 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:47.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:47.797 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:47.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:47.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:47.428+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:47.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:47.428+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:47.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:47.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:47.796+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:48.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:47.796+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:48.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:48.219+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18276 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:48.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:48.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:48.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:48.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:48.464+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:48.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:48.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:48.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:48.676+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:48.676+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:48.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:48.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:48.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:48.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:48.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:48.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:48.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:48.802+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:49.718 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:49.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:49.718 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:49.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:49.718 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:49.441+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:49.718 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:49.441+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:49.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:49.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:49.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:49.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:49.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:49.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:49.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:49.766+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:50.763 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:50.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:50.764 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:50.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:50.764 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:50.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:50.764 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:50.698+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:50.764 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:50.455+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:50.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:50.455+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:50.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:50.455+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:50.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:50.455+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:50.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:50.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:50.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:50.763+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:51.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:51.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:51.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:51.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:51.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:51.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:51.809 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:51.711+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:51.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:51.465+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:51.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:51.465+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:51.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:51.465+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:51.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:51.465+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:51.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:51.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:51.807+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:51.808+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:52.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:52.422+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:52.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:52.422+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:52.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:52.422+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:52.728 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:52.422+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:52.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:52.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:52.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:52.832+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:52.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:52.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:52.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:52.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:53.599 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:53.221+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18358 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:53.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:53.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:53.600 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:53.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:53.600 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:53.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:53.600 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:53.465+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:53.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:53.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:53.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:53.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:53.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:53.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:53.872 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:53.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:53.872+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:53.872+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:53.872+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:53.872+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:54.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:54.484+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:54.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:54.484+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:54.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:54.484+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:54.766 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:54.484+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:54.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:54.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:54.829+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:55.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:54.829+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:55.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:54.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:55.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:54.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:55.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:54.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:55.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:54.765+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:55.782 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:55.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:55.782 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:55.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:55.783 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:55.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:55.783 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:55.739+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:55.783 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:55.530+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:55.783 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:55.530+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:55.783 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:55.530+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:55.783 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:55.530+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:55.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:55.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:55.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:56.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:55.782+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:56.765+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:56.765+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:56.765+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:56.765+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:56.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:56.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:56.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:56.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:56.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:56.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:56.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:56.573+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:56.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:56.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:56.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:56.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:56.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:56.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:56.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:56.764+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:57.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:57.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:57.791+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:57.792+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:57.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:57.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:57.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:57.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:57.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:57.586+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:57.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:57.587+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:57.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:57.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:57.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:57.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:57.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:57.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:57.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:57.746+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:58.614 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:39:58.223+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18441 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:39:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:58.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:58.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:58.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:58.756+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:58.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:58.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:58.614+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:58.614+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:58.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:58.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:58.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:58.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:58.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:58.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:58.774+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:59.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:59.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:59.594+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:39:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:39:59.594+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:39:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:59.723+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:59.723+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:59.723+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:39:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:39:59.723+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:39:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:59.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:59.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:59.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:39:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:39:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:39:59.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:00.884 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:00.728+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:40:00.884 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:00.728+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:40:00.884 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:00.728+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:40:00.884 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:00.728+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:00.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:00.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:00.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:00.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:00.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:00.628+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:00.885 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:00.628+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:00.885 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:00.705+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:00.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:00.705+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:00.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:00.705+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:00.886 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:00.705+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:01.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:40:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:01.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:40:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:01.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:30.776331+0000 (oldest deadline 2024-03-24T11:29:50.776331+0000) 2024-03-24T11:40:02.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:01.729+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:01.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:01.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:01.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:01.675+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:02.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:01.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:02.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:01.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:02.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:01.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:02.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:01.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:03.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:02.735+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:03.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:02.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:03.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:02.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:03.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:02.668+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:02.668+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:02.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:02.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:02.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:02.788+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:03.600 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:03.227+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18523 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:03.783+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:03.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:03.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:03.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:03.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:03.675+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:03.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:03.675+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:03.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:03.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:03.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:03.782+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:04.779+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:04.709+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:04.709+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:04.709+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:04.709+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:04.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:04.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:04.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:04.735+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:05.750+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:05.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:05.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:05.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:05.740+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:06.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:05.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:06.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:05.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:06.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:05.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:34.132403+0000 (oldest deadline 2024-03-24T11:29:54.132403+0000) 2024-03-24T11:40:06.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:05.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:06.702+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:06.739+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:07.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:06.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:07.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:06.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:07.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:06.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:07.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:06.740+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:08.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:07.662+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:08.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:07.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:07.759+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:07.759+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:08.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:07.759+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:08.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:07.760+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:08.230+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18607 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:08.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:08.762+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:08.655+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:08.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:08.720+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:08.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:08.720+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:08.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:08.720+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:08.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:08.721+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:09.609+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:09.798+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:09.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:09.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:09.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:29:41.896011+0000 (oldest deadline 2024-03-24T11:30:01.896011+0000) 2024-03-24T11:40:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:09.715+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:10.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:10.731+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:10.827+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:11.850 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:11.591+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:11.850 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:11.683+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:11.849+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:12.871 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:12.551+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:12.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:12.715+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:13.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:12.870+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:13.505 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:13.233+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18689 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:13.505+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:13.717+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:13.904+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:14.875 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:14.550+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:14.876 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:14.746+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:14.875+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:15.827 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:15.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:15.827 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:15.719+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:15.826+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:16.869 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:16.518+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:16.869 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:16.693+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:17.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:16.868+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:17.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:17.490+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:17.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:17.678+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:18.237 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:17.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:18.517 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:18.236+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18772 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:18.882 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:18.516+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:18.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:18.630+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:18.881+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:19.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:19.507+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:19.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:19.612+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:19.842+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:20.858 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:20.556+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:20.858 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:20.651+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:20.857+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:21.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:21.510+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:21.815 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:21.685+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:21.814+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:22.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:22.658+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:22.809+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:23.535 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:23.238+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18855 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:23.844 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:23.707+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:23.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:23.534+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:24.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:23.844+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:24.888 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:24.490+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:24.888 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:24.739+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:24.887+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:25.751 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:25.531+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:25.751+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:25.898+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:26.869 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:26.779+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:26.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:26.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:26.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:26.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:26.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:26.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:26.870 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:26.575+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:26.868+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:27.821 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:27.796+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:27.821 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:27.547+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:27.821 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:27.547+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:27.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:27.547+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:27.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:27.548+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:27.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:28.588 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:28.241+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 18937 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:28.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:28.817+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:28.828+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:28.587+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:28.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:28.587+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:28.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:28.587+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:28.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:28.587+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:29.795 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:29.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:29.795 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:29.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:29.795 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:29.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:29.795 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:29.542+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:29.844+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:29.794+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:30.882 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:30.750+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:30.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:30.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:30.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:30.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:30.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:30.568+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:30.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:30.568+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:30.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:30.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:30.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:30.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:31.842 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:31.765+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:31.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:31.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:31.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:31.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:31.842 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:31.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:31.843 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:31.523+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:32.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:31.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:31.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:31.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:31.841+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:32.814 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:32.765+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:32.814 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:32.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:32.814 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:32.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:32.814 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:32.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:32.814 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:32.560+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:32.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:32.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:32.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:32.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:33.540 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:33.244+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19019 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:33.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:33.754+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:33.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:33.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:33.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:33.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:33.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:33.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:33.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:33.540+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:33.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:33.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:33.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:33.807+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:34.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:34.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:34.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:34.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:34.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:34.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:34.765 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:34.754+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:34.766 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:34.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:34.766 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:34.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:34.766 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:34.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:34.766 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:34.507+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:34.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:34.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:34.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:34.765+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:35.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:35.526+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:35.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:35.526+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:35.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:35.526+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:35.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:35.526+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:35.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:35.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:35.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:35.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:35.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:35.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:35.802 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:35.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:35.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:35.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:35.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:35.800+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:36.821 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:36.528+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:36.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:36.529+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:36.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:36.529+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:36.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:36.529+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:36.822 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:36.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:36.822 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:36.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:36.823 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:36.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:36.823 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:36.770+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:36.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:36.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:36.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:36.820+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:37.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:37.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:37.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:37.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:37.813+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:37.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:37.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:37.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:37.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:37.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:37.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:37.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:37.578+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:37.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:37.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:37.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:37.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:37.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:37.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:37.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:37.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:38.586 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:38.246+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19102 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:38.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:38.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:38.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:38.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:38.585+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:38.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:38.766+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:38.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:38.766+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:38.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:38.766+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:38.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:38.766+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:38.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:38.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:38.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:38.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:38.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:38.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:38.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:38.782+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:39.614+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:39.614+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:39.614+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:39.614+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:39.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:39.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:39.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:39.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:39.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:39.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:39.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:39.785+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:39.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:39.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:39.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:39.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:39.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:39.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:39.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:39.774+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:40.833+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:40.833+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:40.833+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:40.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:40.833+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:40.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:40.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:40.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:40.796+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:40.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:40.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:40.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:40.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:40.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:40.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:40.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:40.606+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:41.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:41.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:41.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:41.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:41.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:41.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:41.836 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:41.560+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:41.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:41.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:41.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:41.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:41.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:41.800+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:41.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:41.800+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:41.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:41.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:41.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:41.835+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:42.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:42.588+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:42.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:42.588+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:42.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:42.588+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:42.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:42.589+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:42.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:42.753+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:42.753+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:42.753+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:42.754+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:42.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:42.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:42.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:42.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:42.835+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:43.556 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:43.250+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19184 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:43.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:43.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:43.872 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:43.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:43.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:43.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:43.873 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:43.784+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:43.873 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:43.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:43.873 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:43.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:43.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:43.787+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:43.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:43.788+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:43.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:43.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:43.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:43.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:43.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:43.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:43.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:43.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:44.789 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:44.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:44.789 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:44.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:44.789 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:44.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:44.789 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:44.522+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:44.790 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:44.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:44.790 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:44.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:44.790 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:44.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:44.790 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:44.764+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:44.788+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:44.788+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:44.788+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:44.788+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:45.807 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:45.553+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:45.807 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:45.553+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:45.807 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:45.553+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:45.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:45.553+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:45.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:45.756+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:45.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:45.756+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:45.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:45.756+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:45.808 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:45.756+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:45.806+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:45.806+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:45.806+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:45.807+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:46.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:46.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:46.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:46.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:46.795+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:46.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:46.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:46.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:46.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:46.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:46.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:46.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:46.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:46.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:46.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:46.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:46.811+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:47.827 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:47.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:47.828 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:47.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:47.828 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:47.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:47.828 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:47.563+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:47.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:47.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:47.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:47.830+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:47.826+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:47.826+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:47.826+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:47.827+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:48.585 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:48.253+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19267 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:48.848 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:48.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:48.849 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:48.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:48.849 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:48.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:48.849 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:48.585+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:48.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:48.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:48.849 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:48.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:48.850 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:48.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:48.850 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:48.805+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:48.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:48.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:48.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:49.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:48.847+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:49.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:49.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:49.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:49.805+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:49.563+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:49.563+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:49.563+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:49.564+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:50.177 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:49.897+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:50.177 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:49.897+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:50.177 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:49.897+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:50.177 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:49.897+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:50.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:50.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:50.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:50.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:50.882 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:50.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:50.882 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:50.562+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:50.882 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:50.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:50.882 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:50.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:50.882 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:50.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:50.882 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:50.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:50.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:50.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:50.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:51.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:50.881+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:51.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:51.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:51.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:51.816+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:51.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:51.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:51.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:51.573+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:51.904+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:51.904+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:51.904+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:51.904+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:52.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:52.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:52.822+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:52.822+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:52.597+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:52.597+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:52.597+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:52.598+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:53.258 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:52.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:53.258 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:52.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:53.258 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:52.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:53.258 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:52.936+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:53.604 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:53.256+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19349 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:53.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:53.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:53.875 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:53.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:53.875 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:53.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:53.875 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:53.645+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:53.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:53.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:53.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:53.873+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:53.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:53.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:53.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:53.974+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:54.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:54.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:54.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:54.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:54.610+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:54.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:54.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:54.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:54.991+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:54.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:54.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:54.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:54.912+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:55.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:55.614+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:55.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:55.614+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:55.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:55.614+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:55.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:55.615+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:56.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:56.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:56.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:56.016+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:55.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:56.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:55.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:56.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:55.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:56.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:55.872+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:56.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:56.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:56.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:56.836+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:56.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:56.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:56.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:56.612+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.004+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.004+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.004+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.004+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:57.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:57.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:57.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:57.851+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:57.607+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:57.607+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:57.607+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:57.607+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:58.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.985+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:58.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.985+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:58.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.985+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:58.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:57.985+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:58.578 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:40:58.259+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19432 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:40:58.884 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:58.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:58.884 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:58.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:58.885 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:58.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:58.885 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:58.578+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:40:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:40:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:58.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:58.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:58.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:58.884+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:59.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:59.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:59.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:40:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:40:59.622+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:40:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:59.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:59.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:59.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:40:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:40:59.850+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:40:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:40:59.989+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:01.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:00.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:01.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:00.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:01.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:00.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:01.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:00.670+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:01.032 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:00.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:01.032 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:00.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:01.032 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:00.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:01.032 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:00.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:01.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:01.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:01.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:01.030+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:01.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:01.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:01.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:01.818+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:01.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:01.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:01.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:01.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:01.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:01.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:01.636+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:02.304 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:02.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:02.304 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:02.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:02.304 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:02.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:02.304 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:02.046+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:02.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:02.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:02.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:02.638+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:02.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:02.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:02.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:02.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:03.383 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:03.262+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19513 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:03.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:03.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:03.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:03.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:03.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:03.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:03.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:03.049+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:04.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:03.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:04.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:03.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:04.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:03.684+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:04.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:03.685+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:04.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:03.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:04.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:03.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:04.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:03.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:04.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:03.789+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.008+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:04.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.008+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:04.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.008+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:04.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.008+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:04.963 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:04.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:04.963 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:04.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:04.964 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:04.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:04.964 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:04.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:04.964 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:04.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:04.964 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:04.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:04.964 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:04.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:04.964 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:04.800+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:05.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:05.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:05.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:05.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:04.963+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:06.007 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:05.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:06.007 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:05.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:06.007 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:05.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:06.007 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:05.694+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:06.007 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:05.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:06.008 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:05.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:06.008 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:05.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:06.008 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:05.837+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:06.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:06.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:06.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:06.006+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:07.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:06.707+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:07.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:06.707+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:07.037 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:06.707+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:07.037 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:06.708+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:07.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:06.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:07.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:06.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:07.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:06.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:07.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:06.879+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.036+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:07.990 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:07.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:07.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:07.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:07.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:07.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:07.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:07.659+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:07.991 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:07.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:07.991 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:07.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:07.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:07.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:07.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:07.843+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:08.267 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:08.268 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:08.268 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.989+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:08.268 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:07.990+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:08.265+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19595 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:08.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:08.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:08.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:08.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:08.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:08.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:08.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:08.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:08.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:08.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:08.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:08.642+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:08.976+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:08.976+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:08.976+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:08.976+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:09.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:09.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:09.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:09.611+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:09.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:09.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:09.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:09.833+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:09.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:09.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:09.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:09.999+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:10.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:10.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:10.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:10.625+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:10.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:10.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:10.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:10.805+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:11.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:11.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:11.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:11.020+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:11.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:11.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:11.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:11.606+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:11.762+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:11.762+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:11.762+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:11.762+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:12.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:12.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:12.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:12.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:12.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:12.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:12.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:12.642+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:12.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:12.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:12.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:12.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:13.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:13.268+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19677 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:13.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:13.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:13.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:13.042+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:14.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:13.656+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:14.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:13.656+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:14.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:13.656+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:14.077 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:13.656+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:14.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:13.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:14.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:13.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:14.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:13.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:14.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:13.768+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:14.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:14.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:14.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:14.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:14.076+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:14.608+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:14.608+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:14.608+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:14.608+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:14.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:14.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:14.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:14.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:14.782+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.036+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:15.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:15.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:15.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:15.610+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:15.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:15.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:15.795+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:15.795+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:15.999+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:16.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:16.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:16.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:16.606+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:16.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:16.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:16.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:16.789+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.012+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:17.964 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:17.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:17.964 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:17.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:17.965 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:17.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:17.965 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:17.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:17.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:17.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:17.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:17.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:17.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:17.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:17.965 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:17.751+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:18.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:18.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:18.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:18.273 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:17.963+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:18.602 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:18.270+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19761 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:18.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:18.601+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:18.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:18.601+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:18.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:18.601+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:18.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:18.601+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:18.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:18.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:18.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:18.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:18.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:18.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:18.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:18.784+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:18.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:18.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:18.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:18.933+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:19.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:19.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:19.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:19.575+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:19.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:19.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:19.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:19.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:19.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:19.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:19.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:19.971+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:20.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:20.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:20.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:20.584+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:20.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:20.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:20.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:20.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:20.995+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:20.995+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:20.995+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:20.995+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:21.843 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:21.545+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:21.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:21.545+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:21.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:21.545+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:21.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:21.545+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:21.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:21.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:21.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:21.997+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:21.842+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:21.842+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:21.842+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:21.843+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:22.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:22.556+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:22.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:22.556+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:22.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:22.556+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:22.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:22.557+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:22.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:22.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:22.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:22.970+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:22.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:22.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:22.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:22.864+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:23.547 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:23.274+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19845 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:23.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:23.546+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:23.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:23.546+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:23.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:23.546+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:23.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:23.547+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:23.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:23.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:23.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:24.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:23.872+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:24.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:23.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:24.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:23.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:24.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:23.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:24.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:23.993+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:24.856 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:24.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:24.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:24.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:24.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:24.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:24.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:24.535+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:24.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:24.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:24.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:24.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:24.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:24.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:24.948+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:24.949+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:25.819 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:25.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:25.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:25.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:25.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:25.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:25.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:25.553+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:25.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:25.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:25.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:25.819+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:25.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:25.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:25.935+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:25.935+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:26.856 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:26.602+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:26.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:26.602+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:26.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:26.602+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:26.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:26.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:26.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:26.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:26.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:26.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:26.949+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:26.949+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:26.949+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:26.949+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:27.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:27.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:27.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:27.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:27.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:27.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:27.868 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:27.572+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:27.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:27.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:27.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:27.866+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:27.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:27.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:27.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:27.934+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:28.399 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:28.277+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 19930 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:28.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:28.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:28.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:28.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:28.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:28.575+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:28.857 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:28.576+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:28.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:28.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:28.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:29.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:28.980+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:28.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:28.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:28.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:28.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:29.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:29.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:29.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:29.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:29.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:29.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:29.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:29.613+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:29.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:29.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:29.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:29.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:30.899 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:30.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:30.900 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:30.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:30.900 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:30.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:30.900 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:30.647+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.980+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:30.980+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:30.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:30.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:30.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:30.899+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:31.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:31.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:31.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:31.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:31.627+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:32.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:32.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:32.009+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:32.009+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:31.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:31.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:32.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:31.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:32.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:31.941+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:32.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:32.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:32.636+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:32.636+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:33.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:33.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:33.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:33.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:33.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:33.052+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:33.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:33.052+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:33.282 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:32.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:33.282 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:32.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:33.282 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:32.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:33.282 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:32.926+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:33.607 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:33.280+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20014 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:33.887 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:33.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:33.888 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:33.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:33.888 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:33.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:33.888 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:33.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:34.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:34.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:34.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:34.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:33.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:33.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:33.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:33.887+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:34.616+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:34.616+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:34.616+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:34.616+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:34.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:34.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:34.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:34.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:35.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:35.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:35.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:35.103+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:35.808 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:35.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:35.809 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:35.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:35.809 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:35.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:35.809 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:35.573+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:36.085 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:35.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:36.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:35.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:36.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:35.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:36.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:35.808+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:36.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:36.084+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:36.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:36.085+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:36.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:36.085+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:36.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:36.085+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:36.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:36.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:36.609+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:36.610+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:36.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:36.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:36.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:36.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:36.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:36.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:36.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:36.789+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:37.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:37.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:37.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:37.119+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:37.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:37.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:37.571+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:37.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:37.571+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:37.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:37.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:37.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:37.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:38.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:38.283+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20099 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:38.098+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:38.098+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:38.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:38.098+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:38.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:38.099+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:38.767 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:38.582+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:38.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:38.582+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:38.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:38.582+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:38.768 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:38.582+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:39.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:38.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:39.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:38.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:38.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:38.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:39.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:39.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:39.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:39.142+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:39.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:39.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:39.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:39.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:39.615+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:39.615+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:39.615+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:39.616+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:40.567 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:40.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:40.567 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:40.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:40.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:40.151+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:40.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:40.151+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:40.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:40.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:40.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:40.567+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:40.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:40.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:40.739+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:40.739+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:41.106+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:41.106+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:41.106+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:41.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:41.107+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:41.565+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:41.565+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:41.565+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:41.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:41.565+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:41.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:41.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:41.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:41.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:42.543 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:42.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:42.543 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:42.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:42.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:42.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:42.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:42.156+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:42.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:42.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:42.815 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:42.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:42.816 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:42.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:42.816 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:42.543+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:42.815+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:42.815+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:42.815+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:42.815+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:43.502 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:43.286+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20183 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:43.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:43.183+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:43.503 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:43.183+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:43.503 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:43.183+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:43.503 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:43.183+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:43.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:43.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:43.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:43.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:43.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:43.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:43.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:43.502+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:44.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:43.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:44.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:43.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:44.140 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:43.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:44.140 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:43.777+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:44.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:44.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:44.139+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:44.139+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:44.751 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:44.464+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:44.751 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:44.464+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:44.751 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:44.464+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:44.751 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:44.464+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:44.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:44.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:44.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:44.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:45.422 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:45.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:45.422 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:45.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:45.422 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:45.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:45.422 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:45.160+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:45.796 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:45.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:45.796 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:45.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:45.796 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:45.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:45.796 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:45.422+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:45.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:45.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:45.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:45.796+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:46.406+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:46.406+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:46.406+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:46.407+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:46.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:46.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:46.178+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:46.178+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:46.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:46.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:46.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:46.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:47.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:47.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:47.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:47.382+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:47.207+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:47.207+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:47.207+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:47.207+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:47.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:47.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:47.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:47.791+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:48.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:48.289+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20268 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:48.366+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:48.366+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:48.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:48.366+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:48.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:48.366+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:48.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:48.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:48.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:48.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:48.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:48.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:48.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:48.226+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:48.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:48.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:48.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:48.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:49.354+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:49.354+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:49.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:49.354+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:49.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:49.354+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:49.191+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:49.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:49.192+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:49.192+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:49.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:49.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:49.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:49.737+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:50.380+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:50.380+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:50.380+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:50.380+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:50.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:50.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:50.160+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:50.160+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:51.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:50.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:51.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:50.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:51.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:50.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:51.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:50.752+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:51.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:51.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:51.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:51.335+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:51.136+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:51.136+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:51.136+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:51.136+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:51.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:51.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:51.773+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:51.773+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:52.318+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:52.318+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:52.318+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:52.319+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:52.163+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:52.163+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:52.163+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:52.163+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:52.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:52.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:52.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:52.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:53.609 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:53.292+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20352 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:53.609 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:53.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:53.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:53.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:53.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:53.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:53.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:53.169+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:53.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:53.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:53.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:53.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:53.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:53.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:53.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:53.335+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:53.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:53.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:53.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:53.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:53.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:54.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:54.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:54.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:54.347+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:54.132+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:54.132+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:54.132+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:54.132+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:55.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:54.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:55.094 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:54.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:55.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:54.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:55.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:54.780+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:55.323+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:55.323+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:55.323+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:55.323+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:55.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:55.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:55.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:55.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:55.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:55.094+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:55.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:55.094+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:56.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:55.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:56.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:55.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:56.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:55.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:56.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:55.803+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:56.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:56.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:56.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:56.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:56.350 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:56.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:56.350 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:56.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:56.349+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:56.349+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:56.349+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:56.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:56.349+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:57.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:56.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:57.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:56.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:57.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:56.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:57.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:56.809+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:57.334+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:57.334+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:57.334+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:57.334+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:57.114+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:57.114+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:57.114+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:57.114+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:58.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:57.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:58.079 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:57.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:58.079 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:57.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:58.079 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:57.784+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:58.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:41:58.294+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20437 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:41:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:58.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:58.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:58.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:58.077+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:58.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:58.295+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:58.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:58.295+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:58.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:58.295+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:58.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:58.295+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:59.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:59.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:59.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:41:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:41:59.035+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:41:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:58.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:58.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:58.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:41:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:58.807+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:41:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:59.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:59.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:59.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:41:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:41:59.270+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:00.044+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:00.044+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:00.044+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:00.044+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:59.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:59.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:59.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:41:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:41:59.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:00.269+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:00.269+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:00.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:00.269+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:00.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:00.269+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:01.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:00.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:01.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:00.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:01.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:00.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:01.020 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:00.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:01.274 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:01.274 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:01.274 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:01.274 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:01.273+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:01.273+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:01.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:01.273+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:01.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:01.273+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:01.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:01.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:01.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:01.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:01.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:01.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:01.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:01.874+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:02.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:02.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:02.306 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:02.306 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:01.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:02.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:02.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:02.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:02.305+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:03.299 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:02.915+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:03.300 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:02.915+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:03.300 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:02.915+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:03.300 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:02.915+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:03.300 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:03.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:03.300 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:03.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:03.300 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:03.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:03.301 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:03.007+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:03.610 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:03.298+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20518 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:03.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:03.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:03.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:03.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:03.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:03.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:03.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:03.304+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:04.280 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:04.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:04.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:04.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:04.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:04.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:04.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:04.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:04.281 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:03.960+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:04.281 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:03.960+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:04.282 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:03.960+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:04.282 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:03.961+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:04.279+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:04.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:04.279+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:04.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:04.279+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:04.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:04.280+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:05.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:05.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:05.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:05.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:05.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:05.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:05.261 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:05.047+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:05.261 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:04.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:05.261 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:04.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:05.261 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:04.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:05.261 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:04.987+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:05.259+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:05.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:05.259+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:05.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:05.259+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:05.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:05.260+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.015+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:06.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:06.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:06.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:06.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:06.018+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:06.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:06.247+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:06.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:06.247+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:06.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:06.247+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:06.396 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:06.247+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:07.281 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:07.062+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:07.281 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:07.062+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:07.281 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:07.062+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:07.281 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:07.062+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:07.282 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:07.282 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:07.282 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:07.282 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:06.990+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:07.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:07.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:07.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:07.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:07.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:07.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:07.280+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:08.239 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:08.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:08.240 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:08.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:08.240 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:08.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:08.240 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:08.069+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:08.240 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:07.976+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:08.240 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:07.976+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:08.241 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:07.976+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:08.241 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:07.976+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:08.301+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20597 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:08.238+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:08.238+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:08.238+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:08.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:08.239+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:09.230 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:09.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:09.230 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:09.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:09.230 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:09.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:09.230 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:09.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:09.231 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:09.023+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:09.231 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:09.023+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:09.231 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:09.023+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:09.231 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:09.023+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:09.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:09.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:09.229+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:09.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:09.229+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:10.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:10.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:10.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:10.058+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.039+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.039+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:10.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:10.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:10.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:10.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:10.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:10.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:10.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:10.217+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:11.181+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:11.181+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:11.181+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:11.182+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:11.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:11.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:11.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:11.087+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:11.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:11.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:11.397 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:11.398 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:10.997+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:12.037+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:12.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:12.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:12.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:12.038+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:12.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:12.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:12.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:12.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:12.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:12.189+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:12.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:12.189+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:12.082+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:12.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:12.082+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:12.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:12.082+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:12.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:12.083+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:13.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:13.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:13.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.035+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:13.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.035+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:13.307 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:13.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:13.307 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:13.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:13.307 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:13.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:13.307 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:13.133+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:13.307 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:13.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:13.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:13.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:13.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:13.150+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:13.308 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:13.150+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:13.610 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:13.305+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20672 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:14.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:13.988+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:14.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:14.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:14.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:14.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:14.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:14.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:14.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:14.116+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:14.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:14.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:14.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:14.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:14.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:14.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:14.174+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:15.165+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:15.165+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:15.165+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:15.166+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:15.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:15.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:15.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:15.083+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:15.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:14.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:15.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:14.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:15.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:14.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:15.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:14.966+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:16.208 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:16.085+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:16.208 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:16.085+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:16.208 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:16.085+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:16.209 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:16.085+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:16.209 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:15.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:16.209 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:15.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:16.209 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:15.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:16.210 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:15.948+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:16.207+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:16.207+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:16.207+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:16.208+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:17.194 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:17.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:17.194 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:17.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:17.194 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:17.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:17.194 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:17.072+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:17.195 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:16.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:17.195 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:16.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:17.195 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:16.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:17.195 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:16.926+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:17.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:17.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:17.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:17.193+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:18.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:18.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:18.309 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:18.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:18.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:18.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:18.310 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:18.176+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:18.310 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:18.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:18.310 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:18.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:18.310 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:18.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:18.311 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:18.044+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:18.311 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:17.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:18.311 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:17.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:18.311 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:17.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:18.311 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:17.940+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:18.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:18.308+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20748 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:19.186 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:19.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:19.186 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:19.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:19.187 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:19.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:19.187 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:19.086+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:19.187 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:18.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:19.187 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:18.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:19.187 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:18.910+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:19.187 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:18.911+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:19.648 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:19.185+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:19.648 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:19.185+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:19.648 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:19.185+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:19.649 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:19.186+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:20.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:20.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:20.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:20.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:20.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:20.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:20.185 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:20.164+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:20.185 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:20.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:20.186 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:20.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:20.186 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:20.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:20.186 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:20.134+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:20.186 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:19.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:20.186 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:19.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:20.186 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:19.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:20.187 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:19.920+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:21.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:21.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:21.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:21.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:21.153+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:21.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:21.116+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:21.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:21.116+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:21.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:21.116+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:21.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:21.117+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:20.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:21.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:20.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:21.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:20.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:21.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:20.929+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:21.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:21.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:21.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:21.902+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:22.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:22.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:22.143+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:22.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:22.143+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:22.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:22.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:22.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:22.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:22.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:22.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:22.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:22.155+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:23.199 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:23.129+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:23.200 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:23.129+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:23.200 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:23.129+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:23.200 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:23.129+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:23.200 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:22.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:23.200 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:22.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:23.200 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:22.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:23.201 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:22.943+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:23.611 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:23.311+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20823 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:23.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:23.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:23.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:23.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:23.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:23.199+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:23.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:23.199+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:24.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:24.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:24.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:24.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:24.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:24.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:24.192+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:24.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:24.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:24.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:24.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:24.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:24.088+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:23.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:23.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:23.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:23.975+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:25.248 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:25.161+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:25.248 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:25.161+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:25.248 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:25.161+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:25.248 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:25.162+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:25.249 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:25.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:25.249 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:25.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:25.249 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:25.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:25.249 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:25.097+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:25.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:24.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:25.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:24.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:25.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:24.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:25.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:24.967+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:26.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:26.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:26.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:26.124+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:26.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:26.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:26.071+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:26.072+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:25.942+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:26.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:25.942+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:26.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:25.942+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:26.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:25.943+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:26.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:26.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:26.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:26.942+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:27.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:27.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:27.092+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:27.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:27.093+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:27.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:27.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:27.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:27.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:27.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:27.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:27.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:27.039+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:28.294 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:28.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:28.294 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:28.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:28.295 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:28.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:28.295 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:28.053+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:28.295 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:28.055+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:28.295 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:28.055+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:28.295 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:28.055+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:28.295 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:28.055+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:28.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:27.930+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:28.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:27.930+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:28.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:27.930+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:28.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:27.930+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:28.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:28.315+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20902 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:29.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:29.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:29.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:29.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:29.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:29.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:29.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:29.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:29.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:29.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:29.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:29.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:29.086+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:29.396 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:28.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:29.396 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:28.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:29.396 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:28.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:29.396 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:28.929+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:30.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:30.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:30.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:30.037+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:30.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:30.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:30.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:30.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:30.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:30.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:30.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:30.085+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:30.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:29.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:30.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:29.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:30.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:29.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:30.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:29.951+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:31.066+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:31.066+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:31.066+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:31.066+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.030+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:30.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:30.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:30.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:30.948+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:31.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:31.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:31.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:31.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:31.986 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:31.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:31.986 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:31.917+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:32.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:32.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:32.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:32.034+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:31.985+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:33.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:33.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:33.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:33.005+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:33.029+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:33.029+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:33.029+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:33.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:33.029+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:32.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:33.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:32.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:33.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:32.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:33.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:32.892+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:33.612 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:33.318+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 20983 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.017+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.017+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.017+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.017+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:34.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:34.066+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:34.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:34.066+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:34.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:34.066+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:34.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:34.066+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:33.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:34.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:33.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:34.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:33.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:34.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:33.866+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:34.973+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:35.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:35.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:35.056+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:35.056+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:34.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:35.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:34.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:35.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:34.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:35.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:34.852+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:35.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:35.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:35.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:35.953+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:36.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:36.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:36.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:36.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:36.072+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:35.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:36.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:35.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:36.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:35.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:36.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:35.831+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:37.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:36.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:37.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:36.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:37.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:36.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:37.086 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:36.954+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:37.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:36.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:37.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:36.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:37.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:36.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:37.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:36.828+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:37.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:37.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:37.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:37.084+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:38.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:37.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:38.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:37.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:38.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:37.782+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:38.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:37.782+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:37.909+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:37.909+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:37.909+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:37.909+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:38.346 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:38.320+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21068 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:38.347 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:38.083+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:38.347 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:38.083+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:38.347 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:38.083+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:38.347 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:38.084+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:39.042 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:38.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:39.042 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:38.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:39.043 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:38.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:39.043 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:38.780+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:39.043 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:38.888+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:39.043 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:38.888+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:39.043 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:38.888+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:39.043 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:38.888+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:39.041+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:39.041+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:39.041+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:39.041+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:40.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:40.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:40.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:40.050+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:39.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:39.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:39.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:39.857+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:39.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:39.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:40.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:39.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:40.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:39.814+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:40.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:40.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:40.804+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:40.804+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:40.906+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:40.906+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:40.906+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:40.906+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:41.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:41.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:41.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:41.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:41.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:41.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:41.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:41.098+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:41.914+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:41.914+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:41.914+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:42.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:41.914+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:41.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:41.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:41.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:41.807+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:42.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:42.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:42.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:42.147+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:43.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:42.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:43.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:42.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:43.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:42.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:43.124 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:42.914+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:43.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:42.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:43.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:42.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:43.125 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:42.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:43.125 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:42.775+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:43.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:43.324+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21152 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:43.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:43.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:43.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:43.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:43.122+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:44.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:43.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:44.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:43.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:44.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:43.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:44.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:43.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:44.129 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:43.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:44.129 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:43.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:44.129 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:43.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:44.130 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:43.916+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:44.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:44.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:44.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:44.128+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:44.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:44.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:45.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:44.913+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:45.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:44.914+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:45.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:44.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:45.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:44.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:45.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:44.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:45.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:44.716+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:45.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:45.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:45.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:45.101+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:46.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:45.872+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:46.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:45.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:46.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:45.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:46.064 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:45.873+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:46.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:45.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:46.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:45.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:46.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:45.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:46.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:45.690+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:46.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:46.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:46.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:46.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:46.063+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:47.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:46.893+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:47.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:46.893+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:47.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:46.893+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:47.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:46.893+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:47.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:46.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:47.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:46.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:47.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:46.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:47.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:46.672+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:47.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:47.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:47.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:47.072+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:47.637+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:47.637+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:47.637+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:47.638+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:48.330 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:47.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:48.330 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:47.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:48.330 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:47.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:48.331 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:47.935+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:48.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:48.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:48.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:48.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:48.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:48.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:48.331 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:48.105+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:48.618 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:48.328+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21237 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:48.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:48.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:48.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:48.668+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:49.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:49.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:49.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:49.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:49.082+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:49.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:48.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:49.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:48.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:49.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:48.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:49.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:48.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:49.633+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:49.633+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:49.633+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:49.633+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:50.187 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:50.129+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:50.188 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:50.129+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:50.188 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:50.129+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:50.188 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:50.129+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:50.188 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:49.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:50.188 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:49.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:50.188 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:49.933+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:50.189 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:49.933+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:50.624+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:50.624+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:50.624+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:50.624+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:51.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:51.116+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:51.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:51.116+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:51.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:51.116+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:51.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:51.117+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:51.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:50.928+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:51.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:50.928+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:51.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:50.928+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:51.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:50.928+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:51.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:51.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:51.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:51.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:51.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:51.608+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:52.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:52.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:52.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:52.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:52.153+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:51.917+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:51.917+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:51.917+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:51.917+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:52.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:52.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:52.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:52.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:52.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:52.578+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:52.876 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:52.578+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:52.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:52.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:52.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:52.875+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:53.457 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:53.331+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21321 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:53.458 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:53.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:53.458 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:53.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:53.458 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:53.163+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:53.458 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:53.164+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:53.856 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:53.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:53.856 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:53.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:53.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:53.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:53.857 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:53.609+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:53.855+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:53.855+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:53.855+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:54.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:53.856+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:54.641 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:54.195+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:54.641 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:54.195+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:54.641 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:54.195+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:54.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:54.195+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:54.823+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:54.823+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:54.823+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:54.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:54.823+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:54.640+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:54.640+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:54.640+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:54.640+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:55.622 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:55.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:55.623 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:55.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:55.623 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:55.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:55.623 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:55.148+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:55.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:55.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:55.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:55.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:55.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:55.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:55.874 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:55.622+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:55.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:55.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:55.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:55.873+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:56.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:56.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:56.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:56.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:56.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:56.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:56.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:56.175+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:56.851+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:56.851+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:56.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:56.851+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:56.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:56.851+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:56.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:56.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:56.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:56.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:56.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:56.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:56.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:56.611+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:57.597 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:57.221+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:57.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:57.221+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:57.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:57.221+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:57.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:57.222+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:57.854 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:57.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:57.854 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:57.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:57.854 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:57.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:57.854 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:57.597+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:57.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:57.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:57.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:57.853+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:58.463 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:42:58.334+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21406 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:42:58.464 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:58.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:58.464 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:58.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:58.464 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:58.175+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:58.464 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:58.175+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:58.850 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:58.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:58.850 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:58.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:58.851 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:58.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:58.851 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:58.622+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:58.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:58.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:58.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:59.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:58.850+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:59.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:59.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:59.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:59.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:59.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:59.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:42:59.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:42:59.205+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:42:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:59.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:59.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:59.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:59.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:42:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:42:59.830+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:42:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:59.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:59.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:59.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:42:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:42:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:42:59.611+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:00.588 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:00.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:00.588 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:00.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:00.588 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:00.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:00.589 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:00.193+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:00.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:00.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:00.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:00.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:00.797+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:00.587+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:00.587+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:00.587+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:00.587+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:01.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:01.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:01.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:01.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:01.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:01.173+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:01.549 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:01.173+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:01.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:01.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:01.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:01.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:01.797+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:01.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:01.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:01.548+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:01.548+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:02.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:02.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:02.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:02.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:02.192+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:02.752+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:02.752+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:02.752+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:02.752+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:02.552+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:02.552+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:02.552+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:02.552+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:03.465 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:03.337+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21490 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:03.530 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:03.170+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:03.530 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:03.170+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:03.530 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:03.170+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:03.530 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:03.171+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:03.748 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:03.570+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:03.749 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:03.570+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:03.749 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:03.570+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:03.749 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:03.570+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:03.748+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:03.748+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:03.748+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:03.748+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:04.542 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:04.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:04.543 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:04.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:04.543 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:04.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:04.543 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:04.150+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:04.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:04.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:04.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:04.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:04.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:04.743+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:04.542+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:04.542+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:04.542+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:04.542+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:05.498 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:05.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:05.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:05.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:05.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:05.143+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:05.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:05.144+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:05.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:05.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:05.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:05.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:05.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:05.714+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:05.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:05.497+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:05.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:05.497+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:05.896 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:05.497+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:05.896 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:05.497+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:06.522 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:06.167+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:06.523 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:06.167+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:06.523 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:06.167+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:06.523 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:06.167+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:06.681+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:06.681+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:06.681+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:06.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:06.681+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:06.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:06.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:06.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:06.522+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:07.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:07.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:07.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:07.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:07.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:07.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:07.554 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:07.195+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:07.657+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:07.657+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:07.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:07.657+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:07.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:07.658+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:07.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:07.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:07.553+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:07.553+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:08.465 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:08.341+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21575 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:08.466 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:08.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:08.466 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:08.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:08.466 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:08.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:08.466 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:08.193+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:08.644+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:08.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:08.644+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:08.644+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:08.644+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:08.603+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:08.603+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:08.603+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:08.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:08.603+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:09.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:09.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:09.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:09.171+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:09.566+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:09.566+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:09.566+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:09.566+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:09.681+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:09.681+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:09.681+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:09.681+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:10.569 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:10.161+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:10.569 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:10.161+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:10.569 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:10.161+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:10.570 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:10.161+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:10.568+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:10.568+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:10.568+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:10.568+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:10.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:10.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:10.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:10.683+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:11.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:11.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:11.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:11.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:11.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:11.164+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:11.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:11.164+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:11.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:11.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:11.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:11.695+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:11.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:11.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:11.606+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:11.606+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:12.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:12.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:12.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:12.183+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:13.140 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:12.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:13.140 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:12.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:13.141 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:12.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:13.141 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:12.739+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:13.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:12.650+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:13.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:12.650+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:13.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:12.650+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:13.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:12.650+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:13.393 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:13.344+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21658 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:13.140+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:13.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:13.140+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:13.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:13.140+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:13.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:13.140+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:13.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:13.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:13.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:13.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:13.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:13.695+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:13.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:13.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:13.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:13.623+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:14.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:14.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:14.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:14.093+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:14.743 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:14.644+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:14.743 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:14.644+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:14.743 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:14.644+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:14.744 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:14.644+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:15.070 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:14.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:15.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:14.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:15.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:14.743+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:15.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:14.743+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:15.070+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:15.070+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:15.070+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:15.070+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:15.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:15.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:15.615+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:15.615+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:15.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:15.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:15.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:15.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:15.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:15.740+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:15.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:15.740+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:16.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:16.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:16.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:16.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:16.050+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:17.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:16.710+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:17.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:16.710+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:17.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:16.710+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:17.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:16.710+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:17.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:16.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:17.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:16.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:17.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:16.643+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:17.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:16.643+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:17.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:17.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:17.094+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:17.094+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:18.091 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:17.686+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:18.091 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:17.686+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:18.091 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:17.686+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:18.091 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:17.686+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:18.091 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:17.678+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:18.092 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:17.678+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:18.092 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:17.678+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:18.092 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:17.678+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:18.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:18.089+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:18.348 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:18.089+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:18.349 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:18.089+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:18.349 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:18.089+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:18.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:18.346+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21744 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:18.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:18.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:18.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:18.714+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:18.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:18.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:18.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:18.721+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:19.087+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:19.087+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:19.087+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:19.088+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:20.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:19.687+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:20.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:19.687+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:20.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:19.687+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:20.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:19.687+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:20.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:19.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:20.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:19.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:20.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:19.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:20.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:19.751+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:20.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:20.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:20.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:20.084+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:20.800 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:20.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:20.800 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:20.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:20.800 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:20.714+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:20.801 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:20.714+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:21.110 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:20.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:21.110 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:20.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:21.110 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:20.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:21.110 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:20.799+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:21.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:21.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:21.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:21.109+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:22.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:21.702+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:22.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:21.702+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:22.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:21.702+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:22.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:21.702+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:22.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:21.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:22.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:21.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:22.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:21.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:22.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:21.789+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:22.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:22.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:22.071+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:22.072+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:23.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:22.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:23.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:22.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:23.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:22.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:23.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:22.823+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:23.070 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:22.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:23.070 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:22.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:23.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:22.706+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:23.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:22.707+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:23.352 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:23.067+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:23.352 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:23.067+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:23.352 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:23.067+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:23.352 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:23.068+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:23.616 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:23.350+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21827 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:23.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:23.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:23.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:23.723+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:23.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:23.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:23.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:23.840+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:24.083+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:24.083+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:24.083+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:24.083+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:25.066 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:24.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:25.066 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:24.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:25.066 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:24.722+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:25.066 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:24.722+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:25.067 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:24.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:25.067 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:24.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:25.067 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:24.849+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:25.067 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:24.850+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:25.065+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:25.065+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:25.065+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:25.065+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:26.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:25.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:26.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:25.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:26.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:25.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:26.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:25.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:26.062 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:25.752+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:26.062 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:25.752+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:26.062 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:25.752+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:26.062 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:25.753+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:26.060+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:26.060+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:26.060+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:26.061+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:27.066 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:26.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:27.066 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:26.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:27.066 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:26.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:27.066 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:26.844+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:27.066 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:26.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:27.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:26.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:27.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:26.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:27.067 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:26.775+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:27.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:27.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:27.064+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:27.065+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:27.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:27.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:27.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:27.798+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:27.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:27.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:27.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:27.883+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:28.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:28.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:28.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:28.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:28.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:28.030+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:28.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:28.030+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:28.479 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:28.354+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21913 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:28.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:28.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:28.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:28.830+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:28.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:28.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:28.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:28.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:29.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:29.068+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:29.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:29.068+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:29.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:29.068+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:29.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:29.069+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:30.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:29.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:30.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:29.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:30.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:29.941+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:30.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:29.941+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:30.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:29.850+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:30.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:29.850+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:30.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:29.850+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:30.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:29.851+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:30.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:30.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:30.109+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:30.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:30.110+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:31.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:30.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:31.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:30.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:31.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:30.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:31.137 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:30.923+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:31.138 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:30.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:31.138 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:30.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:31.138 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:30.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:31.138 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:30.809+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:31.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:31.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:31.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:31.136+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:31.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:31.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:31.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:31.812+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:31.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:31.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:31.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:31.880+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:32.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:32.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:32.148+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:32.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:32.149+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:33.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:32.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:33.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:32.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:33.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:32.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:33.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:32.831+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:33.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:32.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:33.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:32.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:33.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:32.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:33.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:32.878+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:33.358 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:33.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:33.359 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:33.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:33.359 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:33.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:33.359 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:33.101+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:33.616 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:33.357+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 21996 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:34.108+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:34.108+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:34.108+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:34.108+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:33.871+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:33.871+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:33.871+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:34.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:33.871+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:33.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:34.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:33.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:34.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:33.886+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:34.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:33.886+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:35.125 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:34.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:35.125 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:34.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:35.125 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:34.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:35.125 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:34.822+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:35.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:34.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:35.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:34.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:35.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:34.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:35.126 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:34.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:35.124+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:35.124+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:35.124+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:35.124+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:36.140 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:35.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:35.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:35.810+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:35.810+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:35.801+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:35.801+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:35.801+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:36.141 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:35.802+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:36.139+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:36.139+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:36.139+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:36.140+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:37.138 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:36.755+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:37.138 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:36.755+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:37.139 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:36.755+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:37.139 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:36.756+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:37.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:36.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:37.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:36.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:37.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:36.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:37.139 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:36.828+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:37.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:37.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:37.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:37.137+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:38.124 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:37.731+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:38.124 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:37.731+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:38.125 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:37.731+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:38.125 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:37.731+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:38.125 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:37.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:38.125 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:37.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:38.125 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:37.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:38.125 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:37.834+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:38.383 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:38.361+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22081 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:38.384 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:38.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:38.384 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:38.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:38.384 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:38.123+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:38.384 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:38.124+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:39.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:38.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:39.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:38.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:39.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:38.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:39.112 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:38.763+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:39.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:38.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:39.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:38.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:39.113 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:38.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:39.113 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:38.786+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:39.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:39.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:39.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:39.111+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:40.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:39.730+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:40.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:39.730+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:40.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:39.730+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:40.129 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:39.730+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:40.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:39.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:40.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:39.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:40.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:39.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:40.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:39.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:40.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:40.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:40.127+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:40.128+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:40.737+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:40.737+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:41.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:40.737+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:40.738+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:40.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:40.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:40.791+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:40.791+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:41.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:41.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:41.155+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:41.156+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:42.048 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:41.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:42.048 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:41.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:42.049 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:41.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:42.049 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:41.695+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:42.049 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:41.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:42.049 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:41.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:42.049 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:41.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:42.049 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:41.797+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:42.201+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:42.201+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:42.201+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:42.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:42.201+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:42.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:42.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:42.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:42.723+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:42.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:42.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:42.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:42.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:43.492 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:43.364+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22164 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:43.493 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:43.201+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:43.493 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:43.201+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:43.493 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:43.201+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:43.493 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:43.201+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:43.743 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:43.697+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:43.743 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:43.697+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:43.744 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:43.697+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:43.744 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:43.697+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:43.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:43.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:43.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:43.743+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:44.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:44.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:44.194+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:44.195+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:45.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:44.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:45.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:44.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:45.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:44.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:45.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:44.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:45.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:44.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:45.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:44.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:45.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:44.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:45.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:44.724+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:45.236+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:45.236+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:45.236+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:45.236+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:45.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:45.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:45.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:45.716+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:45.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:45.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:45.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:45.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:46.211+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:46.211+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:46.211+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:46.212+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:46.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:46.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:46.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:46.738+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:46.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:46.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:46.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:46.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:47.209+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:47.209+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:47.209+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:47.209+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:47.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:47.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:47.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:47.757+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:47.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:47.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:47.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:47.777+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:48.494 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:48.368+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22250 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:48.495 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:48.186+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:48.495 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:48.186+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:48.495 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:48.186+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:48.495 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:48.186+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:48.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:48.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:48.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:48.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:49.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:48.779+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:49.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:48.779+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:49.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:48.779+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:49.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:48.779+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:49.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:49.683 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:49.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:49.683 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:49.192+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:49.683 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:49.193+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:49.827+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:49.827+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:49.827+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:50.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:49.827+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:49.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:49.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:49.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:49.807+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:50.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:50.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:50.178+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:50.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:50.178+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:50.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:50.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:50.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:50.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:50.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:50.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:50.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:50.867+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:51.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:51.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:51.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:51.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:51.223+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:51.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:51.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:51.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:51.794+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:51.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:51.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:51.830+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:51.830+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:52.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:52.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:52.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:52.193+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:52.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:52.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:52.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:52.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:52.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:52.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:52.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:52.784+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:53.498 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:53.371+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22333 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:53.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:53.200+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:53.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:53.200+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:53.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:53.200+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:53.499 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:53.200+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:53.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:53.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:54.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:53.778+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:54.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:53.778+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:53.755+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:53.755+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:53.755+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:54.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:53.755+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:54.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:54.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:54.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:54.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:54.217+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:54.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:54.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:54.784+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:55.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:54.784+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:54.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:54.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:54.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:54.723+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:55.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:55.246+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:55.246+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:55.246+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:55.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:55.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:55.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:55.794+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:55.760+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:55.760+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:55.760+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:55.760+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:56.278+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:56.278+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:56.278+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:56.278+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:56.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:56.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:56.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:56.796+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:57.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:56.783+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:57.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:56.783+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:57.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:56.783+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:57.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:56.783+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:57.271+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:57.271+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:57.271+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:57.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:57.271+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:57.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:57.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:57.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:58.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:57.818+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:58.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:57.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:58.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:57.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:58.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:57.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:58.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:57.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:58.401 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:43:58.373+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22418 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:43:58.401 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:58.234+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:58.401 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:58.234+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:58.401 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:58.234+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:58.401 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:58.235+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:58.807+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:58.807+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:58.807+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:43:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:58.807+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:43:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:58.744+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:58.744+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:58.744+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:43:59.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:58.744+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:43:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:59.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:59.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:59.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:43:59.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:43:59.204+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:59.827+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:59.827+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:59.827+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:43:59.827+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:59.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:59.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:59.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:43:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:43:59.697+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:00.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:00.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:00.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:00.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:00.480 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:00.193+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:00.481 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:00.194+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:00.798 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:00.723+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:00.798 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:00.723+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:00.798 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:00.723+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:00.799 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:00.723+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:00.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:00.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:00.797+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:00.797+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:01.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:01.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:01.217+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:01.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:01.217+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:01.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:01.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:01.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:01.743+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:01.808+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:01.808+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:01.808+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:02.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:01.809+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:02.206+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:02.206+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:02.206+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:02.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:02.206+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:02.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:02.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:02.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:02.736+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:02.764+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:02.764+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:02.764+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:02.765+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:03.504 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:03.378+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22501 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:03.505 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:03.213+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:03.505 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:03.213+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:03.505 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:03.213+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:03.505 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:03.213+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:03.810 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:03.718+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:03.811 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:03.718+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:03.811 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:03.718+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:03.811 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:03.718+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:03.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:03.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:04.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:03.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:04.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:03.810+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:04.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:04.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:04.171+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:04.171+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:04.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:04.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:04.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:04.738+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:04.765+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:04.765+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:04.765+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:04.765+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:05.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:05.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:05.152+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:05.152+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:06.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:05.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:06.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:05.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:06.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:05.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:06.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:05.764+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:06.131 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:05.793+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:06.132 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:05.793+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:06.132 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:05.793+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:06.132 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:05.793+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:06.129+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:06.129+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:06.129+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:06.130+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:06.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:06.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:06.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:06.736+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:06.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:06.824+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:06.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:06.824+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:06.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:06.824+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:06.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:06.825+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:07.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:07.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:07.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:07.094+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:08.058 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:07.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:07.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:07.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:07.731+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:07.864+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:07.864+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:07.864+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:08.059 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:07.864+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:08.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:08.057+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:08.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:08.057+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:08.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:08.057+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:08.382 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:08.058+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:08.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:08.381+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22587 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:09.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:08.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:09.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:08.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:09.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:08.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:09.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:08.854+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:09.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:08.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:09.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:08.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:09.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:08.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:09.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:08.766+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:09.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:09.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:09.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:09.086+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:10.132 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:09.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:10.132 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:09.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:10.132 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:09.875+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:10.133 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:09.875+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:10.133 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:09.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:10.133 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:09.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:10.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:09.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:10.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:09.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:10.130+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:10.130+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:10.130+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:10.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:10.131+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:11.113 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:10.836+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:11.114 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:10.836+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:11.114 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:10.836+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:11.114 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:10.836+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:11.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:10.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:11.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:10.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:11.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:10.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:11.115 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:10.803+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:11.112+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:11.112+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:11.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:11.112+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:11.113+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:12.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:11.879+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:12.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:11.879+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:12.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:11.879+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:12.074 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:11.879+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:12.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:11.853+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:12.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:11.853+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:12.075 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:11.853+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:12.075 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:11.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:12.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:12.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:12.072+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:12.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:12.073+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:13.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:12.876+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:13.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:12.876+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:13.097 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:12.876+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:13.097 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:12.877+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:13.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:12.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:13.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:12.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:13.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:12.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:13.097 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:12.871+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:13.385 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:13.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:13.385 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:13.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:13.386 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:13.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:13.386 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:13.096+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:13.384+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22669 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:14.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:13.918+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:14.127 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:13.918+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:14.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:13.918+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:14.128 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:13.918+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:14.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:13.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:14.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:13.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:14.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:13.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:14.129 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:13.864+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:14.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:14.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:14.126+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:14.126+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:15.129 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:14.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:15.130 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:14.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:15.130 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:14.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:15.130 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:14.958+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:15.130 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:14.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:15.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:14.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:15.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:14.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:15.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:14.841+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:15.128+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:15.128+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:15.128+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:15.129+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:15.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:15.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:15.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:15.868+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:16.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:15.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:16.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:15.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:16.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:15.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:16.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:15.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:16.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:16.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:16.157+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:16.158+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:16.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:16.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:16.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:17.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:16.851+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:16.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:16.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:16.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:16.946+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:17.203+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:17.203+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:17.203+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:17.204+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:17.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:17.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:17.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:18.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:17.877+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:17.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:17.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:17.919+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:17.919+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:18.513 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:18.387+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22755 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:18.513 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:18.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:18.514 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:18.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:18.514 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:18.223+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:18.514 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:18.224+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:18.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:18.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:18.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:18.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:19.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:18.893+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:19.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:18.893+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:19.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:18.893+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:19.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:18.893+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:19.198+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:19.198+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:19.198+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:19.198+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:19.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:19.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:19.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:19.816+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:19.871+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:20.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:19.871+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:20.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:19.871+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:20.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:19.872+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:20.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:20.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:20.204+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:20.205+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:20.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:20.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:20.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:20.807+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:20.851+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:20.851+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:20.851+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:20.851+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:21.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:21.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:21.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:21.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:21.180+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:21.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:21.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:21.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:21.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:21.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:21.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:21.985 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:21.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:21.985 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:21.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:21.986 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:21.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:21.986 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:21.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:21.986 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:21.841+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:22.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:22.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:22.183+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:22.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:22.184+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:22.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:22.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:22.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:22.835+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:23.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:22.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:23.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:22.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:23.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:22.755+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:23.146 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:22.755+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:23.515 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:23.391+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22837 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:23.515 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:23.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:23.516 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:23.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:23.516 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:23.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:23.516 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:23.223+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:23.817 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:23.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:23.818 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:23.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:23.818 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:23.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:23.818 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:23.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:23.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:23.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:23.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:24.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:23.817+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:24.591 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:24.198+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:24.591 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:24.198+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:24.592 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:24.198+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:24.592 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:24.199+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:24.861 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:24.724+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:24.862 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:24.724+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:24.862 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:24.724+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:24.862 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:24.724+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:25.136 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:24.861+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:25.137 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:24.861+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:25.137 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:24.861+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:25.137 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:24.861+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:25.191+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:25.191+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:25.191+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:25.191+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:25.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:25.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:25.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:25.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:25.877+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:25.877+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:25.877+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:25.877+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:26.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:26.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:26.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:26.222+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:26.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:26.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:26.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:26.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:26.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:26.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:26.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:27.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:26.840+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:27.239+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:27.239+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:27.239+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:27.239+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:27.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:27.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:27.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:27.730+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:27.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:27.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:27.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:28.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:27.849+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:28.397 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:28.269+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:28.398 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:28.269+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:28.398 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:28.269+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:28.398 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:28.270+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:28.779 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:28.395+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 22923 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:28.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:28.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:28.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:29.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:28.847+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:28.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:28.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:28.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:28.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:29.251+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:29.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:29.251+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:29.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:29.251+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:29.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:29.251+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:29.862+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:29.862+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:30.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:29.862+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:30.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:29.862+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:29.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:29.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:29.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:29.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:30.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:30.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:30.222+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:30.222+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:31.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:30.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:31.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:30.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:31.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:30.813+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:31.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:30.814+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:30.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:30.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:30.775+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:30.776+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:31.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:31.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:31.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:31.257+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:31.839+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:31.839+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:31.839+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:32.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:31.839+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:31.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:31.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:31.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:32.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:31.758+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:32.305+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:32.305+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:32.305+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:32.305+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:32.791+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:32.791+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:32.791+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:33.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:32.792+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:32.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:32.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:32.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:32.794+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:33.621 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:33.398+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23007 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:33.622 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:33.309+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:33.622 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:33.309+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:33.622 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:33.309+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:33.622 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:33.309+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:33.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:33.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:33.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:33.841+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:33.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:33.841+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:33.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:33.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:33.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:33.753+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:34.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:34.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:34.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:34.312+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:34.846+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:34.846+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:35.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:34.846+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:35.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:34.847+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:34.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:34.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:34.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:34.736+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:35.625 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:35.345+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:35.625 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:35.345+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:35.625 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:35.345+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:35.625 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:35.345+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:35.887 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:35.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:35.887 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:35.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:35.888 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:35.767+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:35.888 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:35.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:35.886+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:35.886+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:35.886+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:36.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:35.887+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:36.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:36.393+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:36.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:36.393+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:36.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:36.393+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:36.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:36.393+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:36.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:36.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:36.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:37.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:36.927+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:36.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:36.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:36.777+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:36.777+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:37.732 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:37.426+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:37.732 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:37.426+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:37.733 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:37.426+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:37.733 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:37.426+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:37.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:37.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:37.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:37.937+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:37.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:37.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:37.731+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:37.731+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:38.715 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:38.401+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23093 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:38.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:38.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:38.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:38.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:38.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:38.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:38.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:38.428+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:38.950+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:38.950+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:38.950+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:38.950+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:38.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:38.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:38.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:38.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:38.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:38.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:38.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:38.714+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:39.695 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:39.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:39.695 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:39.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:39.695 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:39.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:39.695 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:39.398+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:39.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:39.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:39.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:39.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:39.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:39.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:39.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:39.693+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:40.391 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:39.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:39.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:39.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:39.956+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:40.389+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:40.389+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:40.389+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:40.389+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:40.943 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:40.681+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:40.943 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:40.681+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:40.943 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:40.681+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:40.943 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:40.682+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:41.360 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:40.942+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:41.360 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:40.942+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:41.360 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:40.942+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:41.360 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:40.942+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:41.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:41.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:41.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:41.359+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:41.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:41.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:41.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:41.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:41.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:41.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:41.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:41.685+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:42.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:41.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:42.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:41.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:42.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:41.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:42.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:41.993+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:42.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:42.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:42.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:42.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:42.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:42.383+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:43.038 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:42.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:43.038 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:42.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:43.038 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:42.721+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:43.039 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:42.721+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:43.372 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:43.372 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:43.372 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.037+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:43.373 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.037+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:43.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:43.405+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23186 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:43.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:43.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:43.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:43.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:43.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:43.372+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:43.997 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:43.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:43.998 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:43.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:43.998 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:43.708+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:43.998 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:43.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.996+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:44.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:43.997+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:44.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:44.413+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:44.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:44.413+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:44.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:44.413+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:44.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:44.414+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:44.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:44.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:44.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:44.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:44.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:44.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:44.994 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:44.706+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:44.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:44.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:44.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:44.993+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:45.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:45.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:45.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:45.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:45.659+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:45.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:45.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:45.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:45.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:45.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:45.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:45.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:45.450+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:46.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:46.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:46.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:46.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:46.012+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:46.665+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:46.665+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:46.665+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:46.665+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:46.484+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:46.484+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:46.484+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:46.485+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.001+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.001+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.001+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.001+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:47.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:47.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:47.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:47.648+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:47.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:47.648+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:47.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:47.436+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:47.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:47.436+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:47.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:47.436+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:47.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:47.437+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.959+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:47.959+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:48.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:48.409+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23272 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:48.463+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:48.463+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:48.463+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:48.464+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:48.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:48.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:48.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:48.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:48.645+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:48.950+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:48.950+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:48.950+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:48.951+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:49.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:49.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:49.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:49.621+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:49.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:49.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:49.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:49.468+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:49.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:49.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:49.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:50.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:49.978+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:50.616+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:50.616+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:50.616+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:50.616+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:50.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:50.512+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:50.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:50.512+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:50.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:50.512+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:50.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:50.513+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:51.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:51.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:51.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:51.023+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:51.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:51.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:51.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:51.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:51.602+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:51.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:51.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:51.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:51.537+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.005+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.005+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:52.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:52.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:52.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:52.612+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:52.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:52.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:52.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:52.535+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.958+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:53.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:52.958+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:53.895 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:53.412+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23355 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:53.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:53.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:53.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:53.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:53.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:53.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:53.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:53.573+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:53.896 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:53.565+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:53.896 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:53.565+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:53.896 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:53.565+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:53.897 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:53.566+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:53.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:53.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:53.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:53.934+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:54.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:54.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:54.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:54.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:54.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:54.537+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:54.528+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:54.528+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:54.528+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:54.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:54.528+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:54.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:54.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:54.974+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:54.975+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:55.776 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:55.562+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:55.776 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:55.562+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:55.776 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:55.562+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:55.777 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:55.562+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:55.777 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:55.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:55.777 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:55.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:55.777 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:55.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:55.777 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:55.507+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:55.938+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:55.938+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:55.938+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:55.938+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:56.551+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:56.551+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:56.551+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:56.551+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:56.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:56.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:56.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:56.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:56.556+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:56.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:56.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:56.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:56.984+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:57.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:57.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:57.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:57.593+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:57.542+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:57.542+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:57.542+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:57.542+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:57.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:57.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:57.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:57.990+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:58.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:44:58.416+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23441 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:44:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:58.506+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:58.506+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:58.506+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:58.506+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:58.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:58.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:58.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:58.641+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:59.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:59.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:59.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:44:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:44:59.020+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:44:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:59.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:59.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:59.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:44:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:44:59.658+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:44:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:59.523+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:59.523+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:59.523+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:44:59.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:44:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:44:59.523+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:00.615+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:00.615+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:00.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:00.615+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:00.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:00.615+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:00.541+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:00.541+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:00.541+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:00.541+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:00.979+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:01.498+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:01.498+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:01.498+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:01.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:01.498+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:01.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:01.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:01.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:01.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:01.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:01.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:01.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:01.619+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:02.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:02.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:02.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:02.014+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:02.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:02.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:02.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:02.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:02.529+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:02.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:02.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:02.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:02.611+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:03.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:03.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:03.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:03.058+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:03.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:03.420+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23524 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:03.600+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:03.600+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:03.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:03.600+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:03.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:03.600+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:03.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:03.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:03.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:03.486+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:04.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:04.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:04.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:04.072+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:04.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:04.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:04.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:04.641+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:04.471+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:04.471+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:04.471+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:04.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:04.471+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:05.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:05.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:05.115+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:05.115+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:05.479+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:05.479+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:05.479+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:05.479+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:05.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:05.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:05.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:05.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:05.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:05.622+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:05.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:05.622+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:06.522 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:06.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:06.523 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:06.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:06.523 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:06.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:06.523 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:06.154+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:06.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:06.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:06.522+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:06.522+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:06.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:06.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:06.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:06.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:06.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:06.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:06.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:06.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:07.562 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:07.145+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:07.563 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:07.145+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:07.563 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:07.145+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:07.563 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:07.145+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:07.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:07.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:07.649+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:07.649+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:07.562+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:07.562+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:07.562+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:07.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:07.562+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:08.425 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:08.163+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:08.425 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:08.163+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:08.425 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:08.163+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:08.426 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:08.163+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:08.696 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:08.423+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23610 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:08.696 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:08.582+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:08.696 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:08.582+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:08.696 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:08.582+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:08.696 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:08.582+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:08.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:08.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:08.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:08.694+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:08.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:08.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:08.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:08.695+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:09.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:09.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:09.154+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:09.154+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:09.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:09.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:09.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:09.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:09.602+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:09.665+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:09.665+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:09.665+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:09.665+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:10.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:10.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:10.109+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:10.109+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:10.623+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:10.623+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:10.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:10.623+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:10.624+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:10.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:10.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:10.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:10.635+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:11.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:11.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:11.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:11.083+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:11.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:11.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:11.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:11.598+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:11.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:11.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:11.645+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:11.645+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:12.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:12.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:12.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:12.107+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:12.591+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:12.591+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:12.591+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:12.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:12.591+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:12.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:12.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:12.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:12.602+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:13.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:13.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:13.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:13.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:13.131+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:13.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:13.427+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23693 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:13.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:13.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:13.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:13.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:13.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:13.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:13.605+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:13.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:13.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:13.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:13.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:13.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:13.599+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:14.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:14.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:14.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:14.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:14.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:14.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:14.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:14.162+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:14.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:14.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:14.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:14.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:14.646+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:14.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:14.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:14.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:14.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:14.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:14.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:14.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:14.638+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:15.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:15.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:15.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:15.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:15.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:15.161+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:15.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:15.161+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:15.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:15.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:15.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:15.695+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:15.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:15.695+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:15.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:15.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:15.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:15.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:15.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:15.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:15.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:15.604+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:16.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:16.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:16.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:16.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:16.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:16.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:16.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:16.157+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:16.705+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:16.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:16.705+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:16.705+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:16.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:16.705+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:16.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:16.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:16.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:16.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:16.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:16.584+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:16.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:16.584+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:17.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:17.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:17.576 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:17.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:17.576 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:17.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:17.576 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:17.187+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:17.729+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:17.729+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:17.729+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:17.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:17.729+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:17.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:17.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:17.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:17.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:17.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:17.573+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:17.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:17.574+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:18.443 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:18.430+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23779 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:18.443 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:18.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:18.444 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:18.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:18.444 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:18.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:18.444 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:18.187+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:18.692+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:18.692+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:18.692+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:18.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:18.692+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:18.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:18.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:18.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:18.543+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:19.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:19.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:19.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:19.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:19.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:19.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:19.544 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:19.232+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:19.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:19.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:19.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:19.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:19.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:19.646+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:19.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:19.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:19.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:19.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:19.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:19.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:19.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:19.543+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:20.554 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:20.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:20.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:20.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:20.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:20.262+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:20.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:20.262+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:20.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:20.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:20.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:20.686+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:20.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:20.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:20.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:20.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:20.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:20.552+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:20.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:20.553+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:21.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:21.258+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:21.515 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:21.258+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:21.515 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:21.258+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:21.515 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:21.258+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:21.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:21.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:21.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:21.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:21.691+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:21.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:21.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:21.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:21.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:21.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:21.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:21.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:21.514+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:22.511+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:22.511+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:22.511+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:22.512+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:22.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:22.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:22.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:22.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:22.293+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:22.709+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:22.709+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:22.709+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:22.709+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:23.559 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:23.435+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23862 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:23.560 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:23.296+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:23.560 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:23.296+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:23.560 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:23.296+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:23.560 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:23.297+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:23.561 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:23.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:23.561 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:23.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:23.561 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:23.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:23.561 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:23.504+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:23.664+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:23.664+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:23.664+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:23.664+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:24.267+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:24.267+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:24.267+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:24.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:24.268+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:24.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:24.472+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:24.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:24.472+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:24.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:24.472+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:24.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:24.472+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:24.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:24.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:24.693+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:25.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:24.693+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:25.237+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:25.237+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:25.237+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:25.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:25.237+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:25.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:25.484+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:25.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:25.484+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:25.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:25.484+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:25.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:25.485+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:25.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:25.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:25.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:26.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:25.732+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:26.496 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:26.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:26.496 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:26.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:26.497 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:26.204+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:26.497 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:26.204+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:26.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:26.452+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:26.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:26.452+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:26.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:26.452+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:26.497 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:26.452+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:26.696+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:26.696+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:26.696+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:26.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:26.696+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:27.251+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:27.251+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:27.251+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:27.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:27.251+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:27.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:27.493+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:27.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:27.493+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:27.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:27.493+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:27.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:27.493+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:27.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:27.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:27.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:27.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:28.439 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:28.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:28.440 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:28.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:28.440 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:28.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:28.440 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:28.282+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:28.748 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:28.438+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 23948 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:28.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:28.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:28.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:28.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:28.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:28.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:28.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:28.531+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:28.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:28.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:29.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:28.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:29.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:28.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:29.568 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:29.283+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:29.569 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:29.283+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:29.569 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:29.283+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:29.569 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:29.284+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:29.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:29.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:29.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:29.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:29.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:29.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:29.567+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:29.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:29.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:29.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:29.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:29.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:29.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:30.570 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:30.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:30.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:30.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:30.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:30.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:30.571 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:30.255+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:30.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:30.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:30.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:30.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:30.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:30.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:30.569+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:30.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:30.569+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:30.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:30.569+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:30.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:30.570+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:31.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:31.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:31.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:31.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:31.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:31.275+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:31.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:31.275+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:31.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:31.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:31.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:31.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:31.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:31.730+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:31.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:31.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:31.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:31.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:31.537+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:32.561 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:32.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:32.561 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:32.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:32.562 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:32.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:32.562 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:32.293+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:32.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:32.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:32.560+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:32.561+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:32.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:32.703+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:32.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:32.703+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:32.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:32.703+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:32.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:32.704+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:33.539 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:33.442+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24031 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:33.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:33.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:33.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:33.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:33.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:33.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:33.540 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:33.281+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:33.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:33.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:33.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:33.539+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:33.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:33.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:33.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:33.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:33.725+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:34.555 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:34.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:34.556 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:34.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:34.556 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:34.249+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:34.556 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:34.249+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:34.554+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:34.554+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:34.554+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:34.555+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:34.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:34.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:34.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:34.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:34.762+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:35.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:35.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:35.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:35.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:35.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:35.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:35.538 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:35.280+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:35.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:35.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:35.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:35.537+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:35.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:35.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:35.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:35.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:35.776+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:36.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:36.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:36.525+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:36.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:36.525+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:36.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:36.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:36.294+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:36.294+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:36.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:36.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:36.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:36.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:37.479+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:37.479+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:37.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:37.479+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:37.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:37.480+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:37.285+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:37.285+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:37.285+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:37.286+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:37.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:37.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:37.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:37.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:38.572 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:38.446+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24117 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:38.573 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:38.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:38.573 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:38.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:38.573 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:38.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:38.573 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:38.516+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:38.573 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:38.289+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:38.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:38.289+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:38.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:38.289+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:38.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:38.290+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:38.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:38.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:38.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:38.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:39.556 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:39.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:39.556 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:39.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:39.556 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:39.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:39.557 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:39.286+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:39.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:39.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:39.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:39.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:39.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:39.717+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:39.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:39.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:39.555+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:39.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:39.556+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:40.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:40.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:40.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:40.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:40.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:40.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:40.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:40.292+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:40.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:40.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:40.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:40.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:40.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:40.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:40.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:40.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:40.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:40.513+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:40.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:40.513+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:41.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:41.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:41.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:41.487+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:41.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:41.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:41.340+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:41.340+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:41.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:41.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:41.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:41.766+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:42.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:42.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:42.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:42.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:42.450+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:42.297+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:42.297+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:42.297+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:42.297+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:42.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:42.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:42.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:42.806+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:43.573 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:43.450+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24200 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:43.573 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:43.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:43.573 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:43.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:43.574 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:43.448+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:43.574 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:43.448+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:43.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:43.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:43.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:43.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:43.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:43.281+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:43.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:43.281+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:43.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:43.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:43.833+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:43.833+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:44.466+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:44.466+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:44.466+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:44.466+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:44.290+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:44.290+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:44.290+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:44.290+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:44.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:44.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:44.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:44.877+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:45.499 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:45.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:45.500 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:45.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:45.500 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:45.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:45.500 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:45.242+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:45.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:45.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:45.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:45.499+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:46.279 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:45.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:46.279 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:45.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:46.279 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:45.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:46.279 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:45.920+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:46.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:46.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:46.278+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:46.278+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:46.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:46.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:46.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:46.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:46.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:46.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:46.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:46.467+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:47.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:46.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:47.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:46.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:47.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:46.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:47.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:46.927+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:47.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:47.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:47.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:47.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:47.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:47.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:47.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:47.245+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:47.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:47.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:47.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:47.501+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:48.226 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:47.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:48.227 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:47.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:48.227 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:47.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:48.227 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:47.913+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:48.481 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:48.454+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24286 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:48.482 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:48.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:48.482 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:48.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:48.482 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:48.226+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:48.482 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:48.226+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:48.887 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:48.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:48.887 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:48.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:48.887 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:48.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:48.887 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:48.505+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:48.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:48.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:48.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:48.887+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:49.522 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:49.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:49.522 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:49.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:49.522 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:49.239+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:49.522 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:49.239+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:49.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:49.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:49.521+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:49.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:49.521+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:50.201 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:49.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:50.201 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:49.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:50.201 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:49.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:50.201 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:49.907+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:50.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:50.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:50.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:50.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:50.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:50.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:50.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:50.225+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:50.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:50.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:50.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:50.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:50.534+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:51.180 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:50.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:51.181 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:50.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:51.181 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:50.927+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:51.181 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:50.927+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:51.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:51.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:51.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:51.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:51.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:51.179+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:51.535 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:51.180+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:51.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:51.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:51.534+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:51.534+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:52.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:51.950+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:52.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:51.950+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:52.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:51.950+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:52.208 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:51.950+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:52.519 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:52.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:52.519 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:52.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:52.519 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:52.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:52.519 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:52.206+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:52.518+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:52.518+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:52.518+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:52.518+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:53.198+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:53.198+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:53.198+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:53.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:53.198+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:52.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:52.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:52.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:52.986+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:53.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:53.457+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24369 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:53.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:53.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:53.480+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:53.480+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:54.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:54.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:54.169+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:54.169+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:53.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:53.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:53.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:54.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:53.965+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:54.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:54.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:54.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:54.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:54.509+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:55.174+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:55.174+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:55.174+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:55.174+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:54.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:54.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:54.989+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:55.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:54.989+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:55.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:55.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:55.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:55.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:55.501+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:56.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:56.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:56.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:56.197+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:56.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:56.197+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:56.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:55.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:56.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:55.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:56.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:55.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:56.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:55.997+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:56.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:56.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:56.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:56.508+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:57.177+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:57.177+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:57.177+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:57.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:57.177+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:56.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:56.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:56.965+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:56.965+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:57.461+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:57.461+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:57.461+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:57.461+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:58.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:58.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:58.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:58.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:58.141+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:57.942+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:57.942+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:57.942+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:57.942+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:58.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:45:58.460+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24455 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:45:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:58.475+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:58.475+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:58.475+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:58.475+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:59.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:59.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:59.112+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:45:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:45:59.112+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:45:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:58.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:58.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:58.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:45:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:58.966+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:45:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:59.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:59.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:59.428+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:45:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:45:59.428+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:00.202 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:00.064+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:00.202 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:00.064+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:00.202 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:00.064+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:00.202 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:00.064+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:00.202 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:59.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:00.203 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:59.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:00.203 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:59.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:00.203 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:45:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:45:59.938+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:00.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:00.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:00.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:00.460+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:01.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:01.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:01.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:01.022+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:00.960+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:00.960+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:00.960+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:00.960+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:01.738 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:01.419+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:01.738 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:01.419+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:01.738 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:01.419+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:01.739 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:01.419+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:02.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:02.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:02.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:02.046+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:01.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:01.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:01.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:01.992+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:02.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:02.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:02.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:02.467+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:03.360 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.008+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:02.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:02.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:02.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:03.361 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:02.967+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:03.627 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:03.465+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24538 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:03.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:03.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:03.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:03.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:03.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:03.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:03.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:03.436+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.995+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.995+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:04.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.995+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:04.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:03.995+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:04.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:03.983+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:04.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:03.983+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:04.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:03.983+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:04.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:03.983+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:04.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:04.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:04.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:04.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:04.416+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:05.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:05.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:05.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:05.035+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:05.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.016+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:05.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.016+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:05.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.016+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:05.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.016+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:05.427+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:05.427+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:05.427+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:05.427+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.021+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:05.987+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:06.437+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:06.437+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:06.437+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:06.438+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:06.984+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:06.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:06.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:06.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:06.969+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:07.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:07.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:07.421+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:07.422+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:07.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:07.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:07.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:07.988+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:07.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:07.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:07.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:07.932+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:08.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:08.469+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24624 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:08.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:08.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:08.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:08.439+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:08.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:08.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:08.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:08.999+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:08.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:08.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:08.955+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:09.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:08.956+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:09.437+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:09.437+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:09.437+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:09.438+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:10.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.012+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.012+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:09.977+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:09.977+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:09.977+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:09.977+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:10.423+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:10.423+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:10.423+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:10.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:10.424+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:10.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:11.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:10.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:11.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:10.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:11.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:10.962+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:11.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:10.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:11.738 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:11.392+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:11.738 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:11.392+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:11.739 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:11.392+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:11.739 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:11.393+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:12.347 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:11.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:12.347 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:11.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:12.347 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:11.996+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:12.348 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:11.996+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:12.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:12.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:12.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:12.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:12.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:12.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:12.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:12.020+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:12.346+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:12.346+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:12.346+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:12.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:12.347+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:13.357 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:12.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:13.357 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:12.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:13.358 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:12.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:13.358 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:12.956+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:13.358 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:13.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:13.358 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:13.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:13.358 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:13.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:13.358 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:13.027+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:13.628 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:13.472+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24707 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:13.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:13.356+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:13.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:13.356+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:13.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:13.356+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:13.629 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:13.357+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:14.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:13.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:14.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:13.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:14.380 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:13.943+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:14.381 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:13.943+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:14.381 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:14.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:14.381 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:14.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:14.381 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:14.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:14.381 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:14.067+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:14.379+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:14.379+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:14.379+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:14.379+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:15.387 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:14.903+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:15.387 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:14.903+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:15.387 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:14.903+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:15.388 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:14.904+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:15.388 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:15.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:15.388 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:15.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:15.388 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:15.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:15.388 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:15.080+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:15.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:15.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:15.386+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:15.386+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:16.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:15.859+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:16.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:15.859+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:16.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:15.859+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:16.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:15.860+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:16.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:16.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:16.101+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:16.101+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:16.821 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:16.413+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:16.821 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:16.413+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:16.821 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:16.413+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:16.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:16.414+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:17.077 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:16.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:17.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:16.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:17.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:16.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:17.078 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:16.819+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:17.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:17.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:17.077+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:17.077+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:17.830 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:17.401+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:17.830 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:17.401+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:17.830 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:17.401+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:17.830 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:17.402+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:18.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:17.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:18.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:17.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:18.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:17.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:18.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:17.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:18.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:18.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:18.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:18.116+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:18.836 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:18.476+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24792 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:18.837 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:18.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:18.837 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:18.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:18.837 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:18.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:18.837 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:18.409+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:19.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:18.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:19.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:18.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:19.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:18.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:19.120 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:18.835+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:19.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:19.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:19.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:19.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:19.119+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:19.863 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:19.403+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:19.863 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:19.403+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:19.863 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:19.403+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:19.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:19.403+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:20.100+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:20.100+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:20.100+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:20.100+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:19.862+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:19.862+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:19.862+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:19.862+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:20.372+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:20.372+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:20.372+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:20.372+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:21.335 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:20.904+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:21.335 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:20.904+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:21.335 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:20.904+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:21.335 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:20.904+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:21.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:21.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:21.336 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:21.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:21.336 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:21.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:21.336 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:21.090+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:21.333+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:21.333+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:21.333+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:21.334+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:22.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:21.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:22.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:21.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:22.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:21.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:22.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:21.901+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:22.319+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:22.319+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:22.319+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:22.320+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:22.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:22.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:22.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:22.131+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:22.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:23.184 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:22.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:23.185 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:22.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:23.185 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:22.852+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:23.185 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:23.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:23.185 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:23.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:23.185 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:23.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:23.185 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:23.097+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:23.606 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:23.480+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24876 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:23.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:23.303+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:23.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:23.303+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:23.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:23.303+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:23.607 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:23.303+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:24.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:23.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:24.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:23.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:24.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:23.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:24.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:23.878+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:24.326+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:24.326+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:24.326+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:24.326+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:24.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:24.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:24.141+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:24.141+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:25.123 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:24.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:25.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:24.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:25.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:24.883+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:25.124 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:24.883+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:25.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:25.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:25.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:25.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:25.354+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:25.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:25.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:25.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:25.123+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:26.342 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:25.924+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:26.342 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:25.924+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:26.342 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:25.924+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:26.342 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:25.925+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:26.343 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:26.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:26.343 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:26.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:26.343 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:26.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:26.343 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:26.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:26.341+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:26.341+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:26.341+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:26.341+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:27.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:27.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:27.124+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:27.124+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:26.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:26.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:26.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:26.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:27.351+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:27.351+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:27.351+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:27.352+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:28.118 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:27.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:28.118 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:27.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:28.119 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:27.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:28.119 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:27.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:28.117+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:28.117+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:28.117+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:28.118+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:28.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:28.303+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:28.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:28.303+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:28.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:28.303+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:28.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:28.304+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:28.856 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:28.484+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 24961 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:29.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:28.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:29.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:28.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:29.134 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:28.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:29.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:28.855+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:29.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:29.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:29.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:29.312+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:29.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:29.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:29.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:29.134+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:29.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:29.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:29.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:29.843+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:30.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:30.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:30.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:30.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:30.287+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:30.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:30.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:30.156+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:30.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:30.156+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:30.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:30.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:30.877+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:30.877+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:31.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:31.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:31.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:31.291+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:31.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:31.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:31.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:31.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:31.203+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:31.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:31.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:31.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:31.918+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:32.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:32.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:32.245+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:32.245+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:32.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:32.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:32.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:32.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:32.249+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:33.284 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:32.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:33.284 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:32.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:33.284 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:32.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:33.284 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:32.957+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:33.614 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:33.487+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25045 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:33.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:33.283+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:33.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:33.283+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:33.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:33.283+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:33.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:33.284+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:33.615 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:33.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:33.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:33.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:33.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:33.288+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:33.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:33.288+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:34.312 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:33.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:34.313 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:33.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:34.313 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:33.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:34.313 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:33.961+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:34.322+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:34.322+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:34.322+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:34.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:34.323+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:34.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:34.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:34.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:34.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:34.312+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:35.311 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:34.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:35.312 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:34.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:35.312 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:34.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:35.312 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:34.913+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:35.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:35.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:35.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:35.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:35.312+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:35.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:35.313+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:35.311+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:35.311+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:35.311+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:35.311+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:36.331 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:35.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:36.331 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:35.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:36.331 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:35.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:36.331 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:35.962+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:36.330+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:36.330+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:36.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:36.330+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:36.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:36.330+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:36.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:36.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:36.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:36.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:36.352+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:37.367 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:37.367 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:37.367 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:37.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.009+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:37.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:37.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:37.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:37.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:37.374+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:37.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:37.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:37.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:37.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:37.366+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:38.342 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:38.343 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:38.343 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.985+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:38.343 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:37.985+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:38.619 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:38.490+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25130 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:38.619 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:38.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:38.619 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:38.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:38.619 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:38.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:38.620 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:38.416+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:38.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:38.341+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:38.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:38.341+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:38.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:38.341+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:38.620 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:38.341+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:39.318 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:38.939+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:39.321 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:38.939+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:39.321 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:38.939+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:39.321 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:38.939+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:39.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:39.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:39.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:39.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:39.442+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:39.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:39.317+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:39.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:39.317+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:39.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:39.317+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:39.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:39.317+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:39.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:39.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:39.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:39.891+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:40.434+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:40.434+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:40.434+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:40.434+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:40.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:40.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:40.292+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:40.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:40.292+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:40.847+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:40.847+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:40.847+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:41.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:40.847+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:41.385+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:41.385+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:41.385+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:41.386+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:41.244+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:41.244+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:41.244+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:41.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:41.244+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:41.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:41.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:41.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:42.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:41.871+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:42.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:42.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:42.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:42.436+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:42.222+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:42.222+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:42.222+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:42.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:42.223+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:43.204 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:42.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:43.204 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:42.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:43.204 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:42.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:43.204 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:42.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:43.497 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:43.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:43.497 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:43.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:43.497 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:43.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:43.497 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:43.203+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:43.498 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:43.409+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:43.498 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:43.409+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:43.498 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:43.409+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:43.498 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:43.410+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:43.882 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:43.495+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25214 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:43.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:43.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:43.881+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:43.881+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:44.419+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:44.419+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:44.419+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:44.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:44.420+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:44.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:44.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:44.245+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:44.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:44.245+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:44.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:44.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:44.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:44.850+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:45.410+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:45.410+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:45.410+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:45.410+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:45.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:45.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:45.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:45.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:45.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:45.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:45.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:45.210+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:45.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:45.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:45.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:45.804+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:46.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:46.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:46.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:46.417+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:46.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:46.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:46.235+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:46.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:46.236+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:46.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:46.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:46.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:46.844+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:47.459+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:47.459+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:47.459+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:47.459+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:47.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:47.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:47.223+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:47.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:47.224+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:47.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:47.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:47.880+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:47.880+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:48.501 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:48.457+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:48.501 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:48.457+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:48.501 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:48.457+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:48.501 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:48.458+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:48.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:48.209+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:48.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:48.209+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:48.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:48.209+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:48.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:48.209+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:48.852 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:48.499+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25298 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:48.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:49.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:48.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:48.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:49.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:48.851+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:49.422+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:49.422+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:49.422+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:49.423+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:49.228+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:49.228+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:49.228+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:49.229+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:49.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:49.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:49.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:49.814+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:50.411+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:50.411+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:50.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:50.411+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:50.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:50.411+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:50.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:50.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:50.272+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:50.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:50.272+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:50.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:50.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:50.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:50.832+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:51.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:51.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:51.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:51.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:51.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:51.461+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:51.317+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:51.317+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:51.317+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:51.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:51.317+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:51.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:51.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:51.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:51.852+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:52.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:52.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:52.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:52.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:52.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:52.496+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:52.305+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:52.305+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:52.305+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:52.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:52.305+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:52.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:52.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:52.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:52.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:53.530 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:53.504+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25382 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:53.531 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:53.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:53.531 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:53.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:53.531 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:53.273+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:53.531 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:53.273+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:53.532 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:53.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:53.532 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:53.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:53.532 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:53.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:53.532 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:53.503+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:54.245 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:53.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:54.245 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:53.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:54.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:53.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:54.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:53.902+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:54.498 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:54.244+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:54.498 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:54.244+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:54.498 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:54.244+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:54.498 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:54.244+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:54.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:54.497+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:54.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:54.497+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:54.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:54.497+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:54.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:54.497+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:54.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:54.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:54.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:54.865+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:55.531 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:55.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:55.532 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:55.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:55.532 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:55.248+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:55.532 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:55.248+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:55.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:55.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:55.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:55.531+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:56.204 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:55.908+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:56.205 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:55.908+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:56.205 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:55.908+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:56.205 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:55.909+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:56.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:56.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:56.488 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:56.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:56.489 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:56.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:56.489 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:56.204+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:56.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:56.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:56.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:56.488+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:57.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:57.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:57.172+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:57.173+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:56.933+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:56.933+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:56.933+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:56.933+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:57.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:57.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:57.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:57.508+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:58.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:58.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:58.186+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:58.187+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:57.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:57.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:57.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:57.946+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:58.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:46:58.508+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25463 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:46:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:58.548+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:58.548+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:58.548+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:58.548+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:59.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:59.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:59.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:46:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:46:59.173+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:46:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:58.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:58.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:58.931+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:46:59.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:58.931+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:46:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:59.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:59.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:59.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:46:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:46:59.538+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:00.184 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:59.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:00.185 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:59.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:00.185 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:59.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:00.185 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:46:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:46:59.920+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:00.565 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:00.183+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:00.565 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:00.183+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:00.565 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:00.183+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:00.565 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:00.184+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:00.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:00.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:00.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:00.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:00.564+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:00.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:00.564+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:01.159 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:00.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:01.159 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:00.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:01.159 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:00.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:01.159 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:00.897+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:01.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:01.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:01.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:01.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:01.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:01.158+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:01.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:01.158+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:01.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:01.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:01.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:01.605+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:02.206 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:01.893+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:02.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:01.893+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:02.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:01.893+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:02.207 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:01.893+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:02.625 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:02.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:02.625 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:02.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:02.625 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:02.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:02.625 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:02.206+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:02.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:02.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:02.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:02.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:02.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:02.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:02.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:02.624+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:02.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:02.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:02.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:02.876+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:03.513 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:03.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:03.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:03.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:03.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:03.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:03.514 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:03.184+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:03.889 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:03.512+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25547 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:03.889 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:03.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:03.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:03.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:03.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:03.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:03.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:03.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:03.888+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:04.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:03.888+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:04.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:03.888+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:04.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:03.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:04.628 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:04.171+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:04.628 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:04.171+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:04.628 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:04.171+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:04.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:04.171+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:04.627+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:04.627+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:04.627+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:04.627+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:04.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:04.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:04.918+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:04.918+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:05.147+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:05.147+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:05.147+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:05.147+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:05.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:05.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:05.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:05.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:05.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:05.578+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:05.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:05.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:05.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:05.948+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:06.145+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:06.145+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:06.145+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:06.145+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:06.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:06.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:06.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:06.544+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:06.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:06.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:06.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:06.974+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:07.181+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:07.181+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:07.181+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:07.181+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:07.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:07.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:07.514+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:07.515+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:07.994+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:07.994+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:07.994+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:07.994+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:08.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:08.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:08.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:08.134+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:08.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:08.516+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25631 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:08.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:08.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:08.523+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:08.525+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:09.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:09.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:09.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:09.022+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:09.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:09.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:09.133+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:09.133+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:09.493+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:09.493+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:09.493+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:09.493+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:10.029+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:10.029+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:10.029+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:10.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:10.029+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:10.099+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:10.099+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:10.099+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:10.099+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:10.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:10.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:10.450+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:10.450+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:11.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:11.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:11.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:11.073+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:11.081+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:11.081+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:11.081+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:11.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:11.081+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:11.744 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:11.411+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:11.744 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:11.411+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:11.745 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:11.411+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:11.745 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:11.411+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:12.082+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:12.082+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:12.082+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:12.082+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:12.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:12.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:12.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:12.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:12.060+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:12.405+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:12.405+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:12.405+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:12.406+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:13.391 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:13.067+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:13.391 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:13.067+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:13.391 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:13.067+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:13.391 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:13.067+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:13.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:13.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:13.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:13.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:13.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:13.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:13.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:13.090+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:13.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:13.521+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25715 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:13.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:13.390+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:13.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:13.390+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:13.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:13.390+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:13.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:13.390+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:14.365 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:14.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:14.365 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:14.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:14.365 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:14.087+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:14.366 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:14.088+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:14.366 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:14.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:14.366 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:14.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:14.366 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:14.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:14.366 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:14.082+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:14.365+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:14.365+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:14.365+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:14.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:14.365+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:15.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:15.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:15.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:15.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:15.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:15.081+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:15.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:15.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:15.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:15.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:15.116+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:15.407+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:15.407+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:15.407+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:15.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:15.407+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:16.099+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:16.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:16.099+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:16.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:16.099+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:16.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:16.100+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:16.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:16.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:16.097+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:16.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:16.097+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:16.418+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:16.418+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:16.418+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:16.418+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:17.388 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:17.120+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:17.389 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:17.120+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:17.389 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:17.120+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:17.389 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:17.120+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:17.389 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:17.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:17.389 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:17.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:17.389 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:17.068+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:17.390 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:17.068+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:17.387+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:17.387+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:17.387+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:17.387+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:18.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:18.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:18.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:18.128+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.019+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:18.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.019+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:18.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:18.524+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25799 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:18.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:18.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:18.397+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:18.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:18.398+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:19.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:19.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:19.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:19.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:19.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:19.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:19.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:19.107+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:19.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:19.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:19.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:19.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:18.976+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:19.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:19.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:19.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:19.381+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:20.366 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:20.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:20.367 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:20.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:20.367 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:20.138+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:20.367 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:20.138+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:20.367 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:19.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:20.367 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:19.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:20.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:19.990+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:20.368 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:19.990+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:20.365+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:20.365+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:20.365+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:20.365+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:21.334 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:20.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:21.334 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:20.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:21.334 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:20.961+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:21.334 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:20.961+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:21.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:21.098+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:21.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:21.098+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:21.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:21.098+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:21.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:21.099+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:21.332+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:21.332+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:21.332+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:21.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:21.333+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:21.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:21.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:21.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:21.975+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:22.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:22.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:22.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:22.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:22.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:22.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:22.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:22.058+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:22.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:22.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:22.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:22.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:22.348+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:23.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:23.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:23.348 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:23.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:23.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:23.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:23.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:23.034+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:23.349 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:23.349 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:23.349 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.011+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:23.349 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.011+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:23.632 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:23.529+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25883 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:23.633 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:23.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:23.633 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:23.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:23.633 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:23.347+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:23.633 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:23.348+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:24.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:24.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:24.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:24.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:24.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:24.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:24.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:24.032+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:24.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:24.386 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:24.386 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:24.386 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:23.968+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:24.384+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:24.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:24.384+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:24.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:24.384+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:24.647 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:24.384+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:25.051+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:25.051+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:25.051+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:25.051+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:24.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:24.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:24.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:24.928+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:25.405+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:25.405+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:25.405+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:25.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:25.405+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:26.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:25.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:26.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:25.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:26.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:25.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:26.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:25.976+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:26.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:26.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:26.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:26.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:26.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:26.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:26.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:26.044+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:26.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:26.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:26.441+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:26.441+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.029+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:26.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:26.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:26.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:27.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:26.966+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:27.457+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:27.457+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:27.457+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:27.458+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:27.993+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:27.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:27.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:27.926+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:28.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:27.927+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:28.658 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:28.533+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 25967 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:28.659 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:28.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:28.659 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:28.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:28.659 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:28.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:28.659 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:28.443+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:28.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:28.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:28.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:28.989+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:28.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:28.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:28.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:29.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:28.929+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:29.401+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:29.401+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:29.401+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:29.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:29.402+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:29.962+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:29.962+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:29.962+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:29.963+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:29.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:29.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:29.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:29.964+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:30.394+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:30.394+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:30.394+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:30.394+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:31.357 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:30.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:31.357 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:30.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:31.357 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:30.946+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:31.358 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:30.947+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:31.358 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:30.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:31.358 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:30.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:31.358 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:30.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:31.358 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:30.954+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:31.356+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:31.356+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:31.356+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:31.357+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:31.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:31.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:31.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:31.986+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:32.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:31.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:32.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:31.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:32.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:31.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:32.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:31.902+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:32.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:32.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:32.373+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:32.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:32.373+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:33.033+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:33.033+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:33.033+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:33.033+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:33.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:32.885+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:33.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:32.885+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:33.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:32.885+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:33.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:32.886+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:33.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:33.537+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26051 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:33.415+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:33.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:33.415+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:33.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:33.415+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:33.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:33.415+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:34.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:34.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:34.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:34.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:34.063+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:33.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:33.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:33.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:33.906+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:34.398+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:34.398+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:34.398+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:34.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:34.398+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:35.075+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:35.075+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:35.075+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:35.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:35.075+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:34.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:34.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:34.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:34.915+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:35.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:35.418+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:35.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:35.418+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:35.874 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:35.418+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:35.875 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:35.419+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:36.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:36.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:36.106+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:36.106+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:35.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:35.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:35.873+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:36.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:35.873+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:36.373 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:36.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:36.374 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:36.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:36.374 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:36.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:36.374 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:36.371+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:37.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:37.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:37.060+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:37.060+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:37.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:36.845+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:37.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:36.845+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:37.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:36.845+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:37.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:36.845+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:37.797 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:37.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:37.797 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:37.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:37.798 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:37.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:37.798 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:37.413+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:37.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:37.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:38.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:37.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:38.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:37.796+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:38.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.032+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:38.668 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:38.541+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26136 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:38.668 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:38.398+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:38.668 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:38.398+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:38.668 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:38.398+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:38.668 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:38.399+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:38.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:38.792+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:38.792+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:38.792+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:38.792+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:39.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:39.384 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:39.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:39.385 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:38.983+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:39.383+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:39.383+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:39.383+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:39.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:39.384+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:39.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:39.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:39.966+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:40.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:39.966+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:39.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:40.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:39.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:40.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:39.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:40.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:39.803+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:40.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:40.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:40.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:40.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:40.339+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:40.944+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:40.944+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:40.944+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:40.944+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:40.795+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:40.795+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:40.795+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:41.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:40.795+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:41.313+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:41.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:41.313+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:41.313+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:41.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:41.313+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:41.949 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:41.790+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:41.949 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:41.790+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:41.949 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:41.790+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:41.949 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:41.790+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:42.354 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:41.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:42.354 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:41.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:42.354 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:41.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:42.355 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:41.948+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:42.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:42.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:42.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:42.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:42.353+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:42.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:42.354+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:42.756+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:42.756+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:42.756+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:43.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:42.757+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:42.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:42.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:42.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:43.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:42.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:43.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:43.545+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26220 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:43.388+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:43.388+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:43.388+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:43.389+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:43.954 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:43.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:43.954 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:43.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:43.954 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:43.757+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:43.954 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:43.757+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:43.953+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:43.953+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:43.953+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:44.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:43.953+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:44.726 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:44.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:44.726 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:44.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:44.726 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:44.416+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:44.727 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:44.416+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:44.725+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:44.725+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:44.725+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:45.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:44.725+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:44.923+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:44.923+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:44.923+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:45.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:44.923+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:45.445+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:45.445+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:45.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:45.445+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:45.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:45.446+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:45.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:45.690+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:45.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:45.690+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:45.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:45.690+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:45.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:45.690+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:45.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:45.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:45.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:45.936+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:46.481+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:46.481+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:46.481+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:46.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:46.482+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:46.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:46.707+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:46.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:46.707+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:46.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:46.707+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:46.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:46.708+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:46.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:46.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:46.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:46.938+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:47.527+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:47.527+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:47.527+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:47.527+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:47.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:47.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:47.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:47.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:47.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:47.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:47.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:47.716+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:47.925+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:47.925+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:47.925+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:47.925+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:48.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:48.549+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26303 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:48.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:48.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:48.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:48.763+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:48.763+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:48.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:48.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:48.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:48.537+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:48.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:48.538+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:48.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:49.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:48.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:48.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:48.967+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:49.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:49.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:49.738+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:49.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:49.738+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:49.528+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:49.528+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:49.528+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:49.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:49.528+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:50.209 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:49.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:50.243 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:49.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:50.243 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:49.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:50.243 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:49.918+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:50.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:50.705+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:50.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:50.705+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:50.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:50.705+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:50.881 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:50.706+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:50.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:50.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:50.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:50.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:50.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:50.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:50.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:50.575+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:50.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:50.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:50.887+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:51.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:50.888+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:51.545+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:51.545+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:51.545+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:51.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:51.545+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:51.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:51.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:51.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:51.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:51.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:51.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:51.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:51.670+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:51.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:51.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:51.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:51.921+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:52.519+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:52.519+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:52.519+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:52.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:52.519+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:52.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:52.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:52.667+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:52.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:52.667+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:52.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:52.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:52.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:53.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:52.951+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:53.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:53.553+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26387 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:53.482+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:53.482+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:53.482+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:53.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:53.482+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:53.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:53.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:53.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:53.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:53.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:53.671+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:53.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:53.671+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:53.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:53.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:53.982+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:54.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:53.983+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:54.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:54.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:54.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:54.483+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:54.678+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:54.678+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:54.678+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:54.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:54.679+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:54.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:54.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:54.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:54.951+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:55.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:55.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:55.476+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:55.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:55.476+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:55.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:55.715+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:55.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:55.715+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:55.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:55.715+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:55.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:55.715+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:55.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:55.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:55.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:56.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:55.950+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:56.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:56.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:56.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:56.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:56.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:56.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:56.748 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:56.491+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:56.746+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:56.746+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:56.746+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:57.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:56.746+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:56.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:56.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:56.992+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:56.992+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:57.736 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:57.470+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:57.736 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:57.470+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:57.737 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:57.470+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:57.737 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:57.470+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:58.019 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:57.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:58.019 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:57.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:58.019 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:57.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:58.019 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:57.734+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.018+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.018+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.018+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.018+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:58.685 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:47:58.556+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26471 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:47:58.685 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:58.456+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:58.685 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:58.456+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:58.685 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:58.456+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:58.686 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:58.457+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:58.975 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:58.751+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:58.976 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:58.751+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:58.976 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:58.751+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:58.976 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:58.751+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:47:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:47:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:47:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:47:58.974+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:59.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:59.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:59.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:47:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:47:59.491+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:47:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:59.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:59.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:59.724+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:47:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:47:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:47:59.724+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:00.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:00.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:00.009+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:00.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:00.009+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:00.469+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:00.469+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:00.469+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:00.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:00.469+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:00.679+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:00.679+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:00.679+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:00.679+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:01.043+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:01.043+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:01.043+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:01.043+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:01.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:01.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:01.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:01.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:01.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:01.670+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:01.749 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:01.670+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:01.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:01.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:01.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:01.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:01.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:01.438+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:01.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:01.439+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:02.040+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:02.040+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:02.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:02.040+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:02.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:02.040+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:02.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:02.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:02.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:02.442+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:02.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:02.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:02.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:02.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:02.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:02.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:02.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:02.657+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:03.053+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:03.053+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:03.053+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:03.053+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:03.684 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:03.561+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26555 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:03.684 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:03.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:03.684 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:03.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:03.685 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:03.412+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:03.685 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:03.413+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:04.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:03.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:04.071 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:03.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:04.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:03.683+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:04.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:03.683+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:04.383 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:04.070+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:04.383 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:04.070+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:04.383 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:04.070+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:04.383 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:04.071+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:04.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:04.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:04.382+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:04.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:04.382+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:05.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:04.721+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:05.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:04.721+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:05.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:04.721+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:05.115 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:04.721+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:05.386 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:05.114+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:05.386 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:05.114+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:05.387 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:05.114+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:05.387 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:05.114+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:05.385+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:05.385+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:05.385+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:05.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:05.386+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:06.117 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:05.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:06.117 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:05.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:06.117 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:05.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:06.117 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:05.724+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:06.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:06.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:06.116+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:06.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:06.116+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:06.678 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:06.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:06.678 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:06.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:06.678 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:06.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:06.679 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:06.418+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:07.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:06.677+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:06.677+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:06.677+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:06.677+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:07.141+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:07.141+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:07.141+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:07.141+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:07.717 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:07.425+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:07.717 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:07.425+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:07.718 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:07.425+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:07.718 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:07.426+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:08.094 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:07.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:08.095 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:07.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:08.095 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:07.716+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:08.095 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:07.716+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:08.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:08.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:08.093+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:08.093+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:08.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:08.565+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26640 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:08.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:08.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:08.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:08.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:08.467+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:08.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:08.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:08.704+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:08.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:08.705+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:09.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:09.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:09.143+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:09.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:09.143+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:09.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:09.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:09.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:09.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:09.500+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:09.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:09.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:09.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:09.744+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:10.456 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:10.186+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:10.456 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:10.186+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:10.456 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:10.186+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:10.456 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:10.186+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:10.762 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:10.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:10.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:10.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:10.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:10.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:10.786 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:10.455+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:10.761+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:10.761+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:10.761+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:10.761+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:11.459 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:11.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:11.459 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:11.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:11.459 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:11.197+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:11.459 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:11.197+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:11.749 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:11.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:11.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:11.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:11.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:11.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:11.750 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:11.458+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:11.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:11.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:11.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:11.811+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:12.423+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:12.423+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:12.423+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:12.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:12.424+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:12.238+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:12.238+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:12.238+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:12.238+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:12.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:12.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:12.853+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:12.853+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:13.571 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:13.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:13.571 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:13.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:13.571 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:13.263+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:13.571 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:13.263+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:13.571 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:13.427+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:13.572 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:13.427+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:13.572 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:13.427+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:13.572 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:13.427+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:13.824 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:13.569+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26724 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:13.823+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:13.823+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:13.823+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:13.824+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:14.440+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:14.440+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:14.440+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:14.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:14.440+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:14.230+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:14.230+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:14.230+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:14.230+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:14.845+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:14.845+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:14.845+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:14.846+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:15.474+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:15.474+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:15.474+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:15.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:15.475+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:15.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:15.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:15.260+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:15.261+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:16.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:15.895+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:16.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:15.895+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:16.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:15.895+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:16.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:15.895+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:16.430+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:16.430+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:16.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:16.430+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:16.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:16.430+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:16.294+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:16.294+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:16.294+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:16.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:16.294+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:16.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:16.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:16.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:16.890+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:17.390+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:17.390+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:17.390+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:17.390+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:17.333+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:17.333+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:17.333+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:17.333+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:18.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:17.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:18.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:17.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:18.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:17.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:17.881+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:18.597 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:18.574+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26809 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:18.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:18.414+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:18.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:18.414+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:18.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:18.414+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:18.598 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:18.415+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:18.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:18.343+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:18.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:18.343+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:18.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:18.343+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:18.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:18.343+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:18.886+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:18.886+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:18.886+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:18.887+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:19.378+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:19.378+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:19.378+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:19.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:19.379+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:19.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:19.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:19.368+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:19.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:19.368+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:20.211 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:19.917+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:20.212 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:19.917+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:20.212 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:19.917+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:20.212 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:19.917+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:20.355+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:20.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:20.355+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:20.355+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:20.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:20.355+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:20.349+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:20.349+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:20.349+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:20.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:20.349+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:20.888+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:20.888+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:20.888+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:20.888+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:21.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:21.353+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:21.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:21.353+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:21.601 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:21.353+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:21.601 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:21.353+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:21.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:21.341+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:21.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:21.341+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:21.601 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:21.341+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:21.602 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:21.342+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:22.298 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:21.899+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:22.299 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:21.899+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:22.299 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:21.899+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:22.299 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:21.899+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:22.298+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:22.298+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:22.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:22.298+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:22.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:22.298+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:22.398+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:22.398+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:22.398+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:22.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:22.398+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:22.855+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:22.855+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:22.855+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:22.855+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:23.579 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:23.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:23.580 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:23.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:23.580 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:23.312+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:23.580 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:23.312+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:23.580 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:23.371+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:23.580 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:23.371+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:23.581 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:23.371+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:23.581 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:23.371+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:23.860 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:23.578+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26894 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:23.860+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:23.860+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:23.860+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:24.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:23.860+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:24.310+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:24.310+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:24.310+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:24.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:24.311+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:24.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:24.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:24.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:24.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:24.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:24.407+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:24.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:24.407+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:24.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:24.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:24.890+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:24.890+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:25.317+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:25.317+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:25.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:25.317+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:25.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:25.317+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:25.454+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:25.454+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:25.454+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:25.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:25.454+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:26.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:25.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:26.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:25.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:26.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:25.915+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:26.335 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:25.915+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:26.334+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:26.334+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:26.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:26.334+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:26.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:26.334+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:26.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:26.408+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:26.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:26.408+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:26.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:26.408+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:26.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:26.408+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:27.378 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:26.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:27.379 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:26.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:27.379 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:26.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:27.379 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:26.916+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:27.378+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:27.378+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:27.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:27.378+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:27.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:27.378+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:27.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:27.435+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:27.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:27.435+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:27.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:27.435+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:27.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:27.435+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:28.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:27.921+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:28.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:27.922+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:28.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:27.922+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:28.338 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:27.922+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:28.599 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:28.582+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 26979 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:28.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:28.484+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:28.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:28.484+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:28.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:28.484+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:28.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:28.485+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:28.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:28.336+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:28.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:28.336+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:28.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:28.336+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:28.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:28.337+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:28.891+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:28.891+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:28.891+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:29.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:28.892+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:29.499+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:29.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:29.499+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:29.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:29.499+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:29.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:29.499+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:29.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:29.299+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:29.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:29.299+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:29.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:29.299+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:29.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:29.300+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:29.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:29.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:29.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:30.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:29.881+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:30.540 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:30.282+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:30.541 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:30.282+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:30.541 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:30.282+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:30.541 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:30.283+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:30.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:30.540+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:30.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:30.540+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:30.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:30.540+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:30.816 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:30.540+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:31.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:30.876+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:31.298 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:30.876+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:31.298 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:30.876+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:31.298 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:30.876+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:31.328+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:31.328+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:31.328+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:31.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:31.328+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:31.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:31.501+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:31.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:31.501+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:31.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:31.501+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:31.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:31.501+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:31.894+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:31.894+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:31.894+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:32.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:31.894+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:32.497+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:32.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:32.497+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:32.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:32.497+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:32.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:32.497+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:32.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:32.307+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:32.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:32.307+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:32.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:32.307+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:32.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:32.307+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:33.278 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:32.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:33.279 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:32.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:33.279 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:32.934+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:33.279 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:32.934+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:33.588 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:33.508+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:33.588 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:33.508+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:33.588 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:33.508+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:33.588 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:33.508+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:33.588 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:33.278+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:33.588 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:33.278+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:33.589 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:33.278+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:33.589 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:33.278+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:33.888 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:33.586+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27062 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:33.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:33.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:33.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:34.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:33.983+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:34.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:34.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:34.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:34.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:34.467+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:34.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:34.256+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:34.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:34.256+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:34.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:34.256+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:34.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:34.256+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:35.271 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:34.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:35.271 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:34.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:35.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:34.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:35.272 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:34.983+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:35.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:35.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:35.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:35.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:35.505+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:35.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:35.505+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:35.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:35.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:35.270+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:35.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:35.270+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:36.285 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:36.285 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:36.286 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:36.286 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.030+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:36.284+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:36.284+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:36.284+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:36.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:36.284+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:36.498+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:36.498+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:36.498+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:36.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:36.498+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:37.257 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:37.257 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:37.257 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:37.258 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:36.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:37.255+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:37.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:37.255+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:37.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:37.255+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:37.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:37.256+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:37.500+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:37.500+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:37.500+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:37.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:37.500+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:38.264 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:37.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:38.264 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:37.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:38.264 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:37.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:38.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:37.970+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:38.531 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:38.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:38.531 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:38.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:38.531 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:38.263+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:38.532 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:38.263+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:38.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:38.590+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27147 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:38.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:38.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:38.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:38.529+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:38.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:38.529+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:39.302 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:38.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:39.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:38.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:39.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:38.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:39.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:38.991+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:39.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:39.301+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:39.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:39.301+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:39.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:39.301+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:39.553 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:39.301+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:39.552+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:39.552+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:39.552+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:39.552+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:40.258 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:39.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:40.259 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:39.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:40.259 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:39.983+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:40.259 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:39.983+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:40.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:40.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:40.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:40.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:40.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:40.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:40.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:40.257+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:40.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:40.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:40.598+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:40.598+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:41.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:40.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:41.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:40.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:41.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:40.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:41.281 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:40.955+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:41.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:41.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:41.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:41.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:41.613 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:41.280+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:41.614 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:41.281+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:41.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:41.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:41.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:41.612+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:42.293 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:41.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:42.293 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:41.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:42.293 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:41.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:42.293 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:41.999+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:42.603 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:42.292+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:42.603 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:42.292+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:42.603 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:42.292+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:42.603 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:42.292+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:42.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:42.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:42.602+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:42.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:42.602+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:43.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:43.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:43.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:43.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:43.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:43.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:43.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:43.034+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:43.596 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:43.316+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:43.596 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:43.316+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:43.596 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:43.316+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:43.597 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:43.316+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:43.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:43.595+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27234 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:43.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:43.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:43.612+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:43.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:43.612+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:44.333 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:44.333 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:44.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:44.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.010+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:44.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:44.332+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:44.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:44.332+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:44.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:44.332+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:44.612 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:44.333+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:44.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:44.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:44.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:44.611+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:45.359 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:45.359 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:45.359 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:45.360 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:44.973+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:45.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:45.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:45.358+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:45.359+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:45.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:45.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:45.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:45.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:45.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:45.595+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:45.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:45.596+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:46.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:45.962+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:46.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:45.962+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:46.349 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:45.962+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:46.350 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:45.963+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:46.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:46.348+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:46.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:46.348+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:46.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:46.348+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:46.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:46.349+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:46.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:46.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:46.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:46.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:46.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:46.609+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:47.382 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:46.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:47.382 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:46.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:47.382 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:46.993+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:47.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:46.994+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:47.380+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:47.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:47.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:47.646 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:47.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:47.646 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:47.381+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:47.646 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:47.628+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:47.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:47.628+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:47.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:47.628+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:47.647 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:47.628+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:48.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:48.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:48.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:48.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:48.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:48.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:48.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:48.035+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:48.643 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:48.599+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27319 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:48.383+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:48.383+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:48.383+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:48.384+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:48.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:48.653+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:48.653+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:48.653+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:48.653+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:49.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:49.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:49.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:49.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:49.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:49.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:49.345 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:49.057+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:49.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:49.344+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:49.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:49.344+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:49.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:49.344+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:49.610 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:49.345+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:49.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:49.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:49.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:49.608+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:49.609+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:50.377 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:50.060+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:50.377 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:50.060+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:50.377 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:50.060+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:50.377 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:50.060+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:50.376+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:50.376+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:50.376+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:50.376+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:50.561+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:50.561+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:50.561+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:50.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:50.561+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:51.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:51.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:51.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:51.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:51.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:51.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:51.385 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:51.067+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:51.384+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:51.384+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:51.384+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:51.384+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:51.588+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:51.588+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:51.588+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:51.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:51.588+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:52.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:52.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:52.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:52.081+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:52.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:52.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:52.402+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:52.403+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:52.587+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:52.587+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:52.587+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:52.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:52.587+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:53.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:53.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:53.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:53.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:53.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:53.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:53.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:53.076+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:53.627 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:53.603+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27404 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:53.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:53.375+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:53.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:53.375+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:53.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:53.375+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:53.628 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:53.375+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:53.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:53.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:53.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:53.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:53.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:53.611+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:53.629 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:53.612+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:54.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:54.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:54.339+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:54.339+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:54.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:54.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:54.121+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:54.122+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:54.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:54.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:54.636+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:54.636+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:55.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:55.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:55.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:55.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:55.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:55.131+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:55.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:55.132+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:55.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:55.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:55.381+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:55.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:55.381+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:55.604+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:55.604+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:55.604+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:55.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:55.604+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:56.126+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:56.126+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:56.126+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:56.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:56.126+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:56.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:56.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:56.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:56.418+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:56.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:56.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:56.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:56.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:56.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:56.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:56.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:56.573+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:57.593 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:57.176+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:57.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:57.176+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:57.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:57.176+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:57.594 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:57.176+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:57.594 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:57.393+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:57.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:57.393+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:57.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:57.393+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:57.595 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:57.393+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:57.593+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:57.593+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:57.593+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:57.593+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:58.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:58.362+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:58.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:58.362+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:58.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:58.362+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:58.609 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:58.362+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:58.609 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:58.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:58.609 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:58.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:58.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:58.187+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:58.610 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:58.188+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:58.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:48:58.608+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27486 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:48:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:58.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:58.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:58.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:58.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:58.611+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:59.321 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:59.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:59.322 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:59.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:59.322 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:59.220+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:48:59.322 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:48:59.221+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:48:59.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:59.320+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:59.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:59.320+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:59.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:59.320+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:48:59.611 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:48:59.321+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:48:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:59.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:59.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:59.610+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:48:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:48:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:48:59.610+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:00.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:00.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:00.184+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:00.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:00.185+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:00.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:00.308+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:00.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:00.308+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:00.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:00.308+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:00.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:00.308+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:00.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:00.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:00.646+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:00.646+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:01.536 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:01.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:01.536 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:01.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:01.536 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:01.199+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:01.537 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:01.199+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:01.537 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:01.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:01.537 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:01.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:01.537 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:01.304+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:01.537 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:01.304+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:01.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:01.674+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:01.946 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:01.674+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:01.946 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:01.674+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:01.946 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:01.675+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:02.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:02.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:02.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:02.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:02.224+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:02.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:02.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:02.287+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:02.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:02.287+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:02.656+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:02.656+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:02.656+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:03.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:02.657+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:03.615 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:03.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:03.615 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:03.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:03.615 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:03.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:03.615 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:03.224+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:03.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:03.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:03.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:03.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:03.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:03.257+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:03.616 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:03.257+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:03.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:03.612+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27571 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:03.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:03.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:03.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:03.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:03.639+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:03.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:03.639+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:04.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:04.213+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:04.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:04.214+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:04.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:04.214+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:04.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:04.214+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:04.599 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:04.212+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:04.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:04.212+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:04.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:04.212+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:04.600 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:04.212+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:04.597+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:04.597+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:04.597+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:04.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:04.598+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:05.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:05.228+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:05.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:05.228+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:05.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:05.228+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:05.614 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:05.228+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:05.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:05.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:05.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:05.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:05.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:05.180+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:05.615 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:05.180+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:05.613+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:05.613+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:05.613+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:05.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:05.613+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:06.133+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:06.133+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:06.133+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:06.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:06.134+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:06.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:06.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:06.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:06.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:06.182+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:06.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:06.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:06.618+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:06.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:06.618+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:07.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:07.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:07.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:07.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:07.096+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:07.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:07.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:07.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:07.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:07.144+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:08.108 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:07.665+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:08.108 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:07.665+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:08.109 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:07.665+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:08.109 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:07.665+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:08.108+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:08.108+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:08.108+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:08.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:08.108+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:08.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:08.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:08.189+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:08.189+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:08.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:08.617+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27655 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:08.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:08.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:08.698+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:08.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:08.698+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:09.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:09.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:09.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:09.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:09.093+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:09.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:09.093+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:09.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:09.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:09.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:09.224+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:09.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:09.224+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:10.113 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:09.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:10.113 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:09.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:10.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:09.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:10.114 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:09.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:10.113+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:10.113+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:10.113+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:10.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:10.113+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:10.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:10.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:10.188+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:10.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:10.188+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:11.082 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:10.723+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:11.082 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:10.723+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:11.082 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:10.723+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:11.082 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:10.723+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:11.219+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:11.219+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:11.219+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:11.219+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:11.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:11.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:11.081+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:11.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:11.082+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:12.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:11.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:12.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:11.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:12.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:11.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:12.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:11.737+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:12.216+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:12.216+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:12.216+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:12.216+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:12.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:12.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:12.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:12.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:12.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:12.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:12.395 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:12.086+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:13.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:12.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:13.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:12.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:13.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:12.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:13.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:12.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:13.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:13.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:13.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:13.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:13.051+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:13.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:13.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:13.221+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:13.221+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:13.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:13.621+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27740 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:13.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:13.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:13.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:13.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:13.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:13.828+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:14.097+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:14.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:14.097+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:14.097+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:14.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:14.098+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:14.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:14.190+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:14.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:14.190+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:14.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:14.190+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:14.396 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:14.190+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:15.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:14.821+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:15.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:14.821+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:15.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:14.821+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:15.111 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:14.821+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:15.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:15.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:15.170+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:15.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:15.170+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:15.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:15.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:15.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:15.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:15.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:15.111+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:15.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:15.111+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:16.130 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:16.085+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:16.131 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:16.085+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:16.131 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:16.085+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:16.131 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:16.085+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:16.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:15.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:16.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:15.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:16.131 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:15.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:16.132 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:15.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:16.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:16.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:16.128+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:16.129+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:17.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:17.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:17.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:17.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:17.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:17.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:17.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:17.081+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:17.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:16.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:17.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:16.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:17.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:16.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:17.106 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:16.834+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:17.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:17.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:17.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:17.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:17.100+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:17.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:17.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:17.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:17.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:18.069+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:18.069+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:18.069+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:18.069+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:18.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:18.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:18.122+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:18.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:18.122+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:18.877 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:18.624+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27825 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:19.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:19.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:19.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:19.086+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:18.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:18.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:18.876+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:18.876+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:19.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:19.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:19.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:19.096+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:19.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:19.097+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:20.127 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:19.869+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:20.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:19.869+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:20.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:19.869+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:20.128 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:19.869+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:20.128 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:20.105+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:20.128 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:20.105+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:20.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:20.105+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:20.129 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:20.105+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:20.126+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:20.126+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:20.126+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:20.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:20.126+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:21.077+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:21.077+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:21.077+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:21.077+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:20.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:20.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:20.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:20.840+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:21.146+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:21.146+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:21.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:21.146+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:21.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:21.146+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:21.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:21.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:21.850+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:22.059 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:21.851+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:22.058+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:22.058+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:22.058+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:22.058+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:22.130+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:22.130+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:22.130+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:22.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:22.130+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:23.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:22.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:23.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:22.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:23.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:22.806+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:23.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:22.806+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:23.136 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:23.054+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:23.136 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:23.054+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:23.136 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:23.054+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:23.136 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:23.055+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:23.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:23.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:23.134+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:23.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:23.135+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:23.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:23.629+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27910 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:23.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:23.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:23.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:23.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:23.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:23.836+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:24.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:24.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:24.098+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:24.099+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:24.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:24.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:24.173+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:24.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:24.173+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:25.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:24.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:25.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:24.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:25.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:24.835+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:25.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:24.835+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:25.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:25.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:25.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:25.087+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:25.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:25.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:25.210+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:25.210+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:26.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:25.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:26.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:25.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:26.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:25.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:26.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:25.789+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:26.043+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:26.043+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:26.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:26.043+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:26.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:26.044+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:26.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:26.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:26.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:26.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:26.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:26.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:26.182+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:27.051 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:26.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:27.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:26.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:27.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:26.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:27.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:26.767+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:27.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:27.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:27.050+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:27.051+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:27.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:27.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:27.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:27.232+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.027+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:27.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:27.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:27.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:27.812+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:28.635 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:28.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:28.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:28.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:28.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:28.225+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:28.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:28.225+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:28.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:28.633+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 27994 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:28.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:28.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:28.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:28.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:28.811+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:29.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:29.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:29.202+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:29.203+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:29.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.990+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:29.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.990+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:29.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.990+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:29.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:28.990+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:29.999+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:29.999+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:29.999+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:29.999+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:29.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:29.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:29.786+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:29.786+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:30.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:30.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:30.232+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:30.232+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:31.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:31.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:31.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:31.008+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:30.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:30.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:30.827+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:30.827+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:31.645 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:31.211+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:31.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:31.211+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:31.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:31.211+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:31.646 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:31.212+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:32.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:31.846+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:32.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:31.846+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:32.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:31.846+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:32.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:31.846+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:32.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:32.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:32.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:32.035+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:32.190+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:32.190+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:32.190+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:32.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:32.190+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:33.083 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:32.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:33.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:32.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:33.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:32.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:33.084 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:32.808+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:33.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:33.082+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:33.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:33.082+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:33.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:33.082+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:33.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:33.083+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:33.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:33.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:33.208+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:33.208+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:33.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:33.638+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28078 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:33.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:33.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:33.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:33.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:33.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:33.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:33.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:33.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:34.077+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:34.077+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:34.077+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:34.077+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:34.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:34.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:34.164+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:34.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:34.164+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:35.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:34.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:35.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:34.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:35.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:34.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:35.103 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:34.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:35.102+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:35.102+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:35.102+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:35.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:35.102+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:35.149+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:35.149+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:35.149+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:35.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:35.149+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:36.087 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:36.062+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:36.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:36.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:36.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:36.063+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:36.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:36.063+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:36.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:35.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:36.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:35.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:36.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:35.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:36.089 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:35.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:36.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:36.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:36.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:36.108+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:37.014 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:36.815+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:37.014 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:36.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:37.014 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:36.816+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:37.015 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:36.816+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:37.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:37.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:37.118+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:37.118+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:37.043+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:37.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:37.043+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:37.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:37.043+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:37.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:37.043+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:38.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:37.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:38.086 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:37.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:38.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:37.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:38.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:37.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:38.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:38.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:38.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:38.087 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:38.072+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:38.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:38.085+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:38.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:38.085+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:38.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:38.085+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:38.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:38.086+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:38.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:38.642+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28163 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:38.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:38.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:38.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:38.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:38.781+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:39.068+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:39.068+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:39.068+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:39.068+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:39.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:39.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:39.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:39.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:39.394 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:39.104+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:40.090 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:39.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:40.091 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:39.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:40.091 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:39.807+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:40.091 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:39.808+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:40.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:40.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:40.149+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:40.150+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:40.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:40.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:40.090+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:40.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:40.090+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:41.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:40.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:41.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:40.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:41.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:40.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:41.117 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:40.843+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:41.179+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:41.179+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:41.179+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:41.179+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:41.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:41.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:41.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:41.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:41.116+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:41.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:41.116+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:42.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:41.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:42.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:41.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:42.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:41.834+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:42.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:41.834+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:42.210+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:42.210+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:42.210+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:42.211+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:42.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:42.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:42.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:42.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:42.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:42.069+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:42.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:42.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:42.868+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:42.869+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:43.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:43.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:43.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:43.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:43.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:43.107+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:43.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:43.108+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:43.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:43.215+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:43.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:43.215+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:43.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:43.215+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:43.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:43.216+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:43.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:43.647+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28247 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:44.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:43.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:44.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:43.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:44.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:43.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:44.250 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:43.917+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:44.250 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:44.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:44.250 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:44.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:44.251 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:44.058+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:44.251 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:44.058+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:44.544 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:44.249+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:44.544 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:44.249+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:44.544 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:44.249+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:44.544 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:44.249+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:44.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:44.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:44.892+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:45.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:44.892+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:45.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:45.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:45.103+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:45.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:45.103+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:45.288+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:45.288+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:45.288+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:45.289+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:46.336 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:46.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:46.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:46.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:46.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:46.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:46.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:46.055+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:46.337 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:45.930+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:46.337 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:45.930+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:46.338 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:45.930+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:46.338 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:45.930+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:46.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:46.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:46.335+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:46.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:46.336+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:47.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:47.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:47.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:47.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:47.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:47.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:47.303 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:47.039+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:47.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:46.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:47.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:46.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:47.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:46.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:47.304 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:46.948+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:47.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:47.302+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:47.302+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:47.302+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:47.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:47.302+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:48.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:48.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:48.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:48.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:48.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:48.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:48.260 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:48.022+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:48.260 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:47.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:48.260 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:47.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:48.261 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:47.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:48.261 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:47.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:48.258+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:48.258+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:48.258+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:48.259+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:48.939 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:48.651+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28332 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:49.225 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:49.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:49.225 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:49.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:49.225 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:49.014+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:49.225 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:49.015+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:49.226 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:48.939+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:49.226 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:48.939+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:49.226 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:48.939+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:49.226 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:48.939+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:49.225+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:49.225+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:49.225+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:49.225+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:50.275 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:50.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:50.276 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:50.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:50.276 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:50.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:50.276 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:50.031+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:50.276 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:49.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:50.276 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:49.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:50.277 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:49.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:50.277 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:49.968+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:50.274+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:50.275+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:50.275+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:50.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:50.275+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:51.291 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:51.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:51.292 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:51.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:51.292 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:51.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:51.292 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:51.028+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:51.292 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:50.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:51.293 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:50.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:51.293 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:50.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:51.293 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:50.986+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:51.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:51.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:51.291+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:51.291+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:52.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:51.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:52.045 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:51.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:52.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:51.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:52.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:51.995+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:52.283+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:52.283+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:52.283+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:52.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:52.283+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:52.044+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:52.044+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:52.044+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:52.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:52.045+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:53.333 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:52.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:53.333 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:52.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:53.334 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:52.949+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:53.334 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:52.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:53.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:53.092+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:53.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:53.092+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:53.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:53.092+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:53.334 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:53.092+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:53.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:53.333+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:53.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:53.333+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:53.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:53.333+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:53.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:53.333+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:53.991 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:53.656+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28415 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:54.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:53.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:54.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:53.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:54.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:53.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:54.370 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:53.991+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:54.371 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:54.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:54.371 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:54.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:54.371 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:54.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:54.371 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:54.086+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:54.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:54.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:54.370+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:54.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:54.370+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:55.375 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:55.375 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:55.376 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:55.376 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.006+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:55.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:55.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:55.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:55.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:55.376 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:55.057+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:55.377 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:55.057+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:55.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:55.375+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:55.375+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:55.375+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:55.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:55.375+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:56.372 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:56.372 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:56.372 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.976+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:56.372 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:55.976+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:56.373 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:56.373 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:56.373 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:56.373 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.048+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:56.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:56.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:56.371+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:56.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:56.371+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:56.998+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:56.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:56.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:56.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:56.958+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:57.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:57.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:57.417+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:57.417+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:57.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:57.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:57.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:57.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:57.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:57.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:57.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:57.957+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:58.662 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:58.415+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:58.662 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:58.415+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:58.662 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:58.415+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:58.662 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:58.415+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:58.978 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:49:58.659+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28500 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:49:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:58.999+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:58.999+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:58.999+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:49:59.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:49:58.999+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:49:59.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:58.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:58.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:58.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:49:59.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:49:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:49:58.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:49:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:59.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:59.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:59.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:49:59.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:49:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:49:59.408+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:00.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:00.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:00.004+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:00.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:00.004+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:00.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:00.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:00.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:00.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:00.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:00.025+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:00.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:00.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:00.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:00.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:00.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:00.442+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:00.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:00.442+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:01.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:01.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:01.022+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:01.022+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:01.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:01.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:01.011+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:01.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:01.758 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:01.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:01.758 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:01.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:01.758 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:01.408+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:01.758 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:01.408+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:02.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:02.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:02.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:02.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:02.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:02.003+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:02.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:02.003+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:02.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:02.045+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:02.045+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:02.045+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:02.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:02.046+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:02.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:02.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:02.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:02.459+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.023+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.023+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.023+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.024+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:03.060+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:03.060+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:03.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:03.060+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:03.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:03.060+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:03.789 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:03.665+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28584 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:03.789 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:03.505+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:03.790 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:03.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:03.790 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:03.506+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:03.790 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:03.506+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:04.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:04.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:04.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:04.062 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:03.975+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:04.061+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:04.061+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:04.061+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:04.061+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:04.490+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:04.490+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:04.490+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:04.490+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:04.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:04.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:04.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:05.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:04.987+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:05.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:05.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:05.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:05.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:05.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:05.026+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:05.462+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:05.462+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:05.462+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:05.462+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:06.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:06.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:06.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:06.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:06.037+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:05.998+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:05.998+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:05.998+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:06.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:05.999+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:06.472+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:06.472+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:06.472+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:06.473+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:07.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:07.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:07.053+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:07.054+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.005+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.005+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.005+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:07.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.005+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:07.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:07.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:07.435+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:07.436+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:08.066+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:08.066+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:08.066+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:08.067+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.974+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:08.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:07.974+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:08.794 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:08.670+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28669 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:08.794 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:08.470+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:08.794 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:08.470+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:08.795 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:08.470+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:08.795 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:08.471+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:09.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:08.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:09.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:08.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:09.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:08.947+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:09.106 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:08.947+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:09.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:09.104+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:09.104+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:09.104+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:09.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:09.104+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:09.504+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:09.504+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:09.504+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:09.505+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:10.220 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:09.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:10.220 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:09.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:10.220 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:09.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:10.220 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:09.936+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:10.221 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:10.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:10.221 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:10.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:10.221 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:10.076+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:10.221 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:10.077+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:10.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:10.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:10.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:10.467+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:10.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:10.468+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:11.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:11.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:11.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:11.030+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:10.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:11.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:10.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:11.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:10.956+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:11.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:10.957+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:11.758 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:11.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:11.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:11.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:11.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:11.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:11.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:11.461+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:12.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:11.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:12.047 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:11.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:12.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:11.929+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:12.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:11.929+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:12.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:12.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:12.046+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:12.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:12.047+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:12.499+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:12.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:12.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:12.500+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:12.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:12.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:12.958+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:12.958+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:13.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:13.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:13.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:13.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:13.080+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:13.799 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:13.674+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28753 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:13.799 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:13.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:13.800 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:13.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:13.800 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:13.487+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:13.800 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:13.488+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:14.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:14.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:14.042+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:14.043+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:13.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:13.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:13.957+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:14.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:13.957+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:14.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:14.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:14.500+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:14.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:14.501+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:15.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:14.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:15.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:14.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:15.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:14.962+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:15.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:14.963+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:15.264 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:15.016+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:15.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:15.016+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:15.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:15.016+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:15.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:15.017+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:15.515+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:15.515+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:15.515+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:15.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:15.516+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:15.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:15.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:15.975+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:16.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:15.975+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:16.041+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:16.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:16.041+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:16.041+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:16.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:16.041+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:16.524+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:16.524+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:16.524+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:16.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:16.524+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:16.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:16.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:16.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:16.991+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:17.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:17.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:17.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:17.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:17.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:17.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:17.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:17.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:17.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:17.538+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:17.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:17.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:17.991+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:17.991+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:18.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:18.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:18.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:18.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:18.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:18.805 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:18.679+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28838 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:18.805 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:18.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:18.805 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:18.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:18.805 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:18.531+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:18.805 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:18.531+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:19.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:19.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:19.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:19.049+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:18.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:18.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:18.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:18.987+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:19.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:19.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:19.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:19.574+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:20.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:20.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:20.007+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:20.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:20.007+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:20.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:20.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:20.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:20.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:20.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:20.050+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:20.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:20.050+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:20.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:20.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:20.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:20.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:20.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:21.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:21.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:21.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:21.017+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:21.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:21.017+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:21.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:21.051+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:21.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:21.051+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:21.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:21.051+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:21.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:21.051+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:21.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:21.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:21.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:21.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:21.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:21.576+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:21.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:21.577+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:22.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:22.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:22.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:22.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:22.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:22.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:22.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:22.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:22.057+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:22.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:22.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:22.532+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:22.532+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:23.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:23.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:23.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:23.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:23.033+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:23.110+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:23.110+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:23.110+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:23.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:23.110+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:23.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:23.683+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 28922 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:23.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:23.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:23.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:23.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:23.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:23.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:23.566+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:24.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:24.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:24.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:24.152+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:24.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:24.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:24.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:24.032+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:24.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:24.032+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:24.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:24.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:24.541+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:24.542+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.012+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.012+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.012+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.012+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:25.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:25.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:25.185+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:25.185+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:25.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:25.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:25.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:25.578+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.986+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:25.986+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:26.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:26.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:26.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:26.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:26.162+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:26.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:26.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:26.561+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:26.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:26.561+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:26.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:26.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:26.959+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:26.959+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:27.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:27.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:27.140+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:27.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:27.140+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:27.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:27.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:27.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:27.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:27.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:27.979+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:27.979+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:27.979+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:27.980+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:28.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:28.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:28.152+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:28.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:28.152+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:28.689 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:28.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:28.690 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:28.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:28.690 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:28.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:28.690 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:28.580+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:28.964 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:28.688+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29007 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:28.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:28.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:28.964+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:28.964+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:29.198+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:29.198+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:29.198+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:29.198+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:29.664 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:29.544+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:29.664 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:29.544+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:29.664 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:29.544+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:29.664 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:29.544+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:30.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:30.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:30.203+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:30.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:30.203+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:29.973+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:29.973+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:29.973+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:30.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:29.973+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:30.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:30.498+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:30.498+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:30.498+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:30.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:30.499+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:31.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:31.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:31.182+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:31.182+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:30.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:30.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:30.995+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:30.995+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:31.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:31.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:31.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:31.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:31.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:31.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:31.761 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:31.456+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.000+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:32.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.000+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:32.450 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:32.157+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:32.450 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:32.157+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:32.450 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:32.157+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:32.450 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:32.157+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:32.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:32.449+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:32.449+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:32.449+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:32.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:32.449+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:33.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:33.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:33.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.968+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:33.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:32.968+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:33.137+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:33.137+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:33.137+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:33.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:33.137+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:33.716 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:33.692+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29091 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:33.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:33.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:33.716 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:33.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:33.717 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:33.455+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:33.717 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:33.455+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:34.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:34.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:34.125+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:34.125+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:33.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:33.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:33.967+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:34.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:33.967+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:34.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:34.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:34.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:34.491+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:34.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:34.491+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:35.113+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:35.113+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:35.113+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:35.113+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:34.998+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:34.998+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:34.998+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:35.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:34.998+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:35.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:35.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:35.492+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:35.492+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:36.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:36.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:36.083+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:36.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:36.083+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:35.994+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:35.994+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:35.994+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:36.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:35.994+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:36.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:36.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:36.458+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:36.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:36.458+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:37.114+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:37.114+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:37.114+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:37.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:37.114+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:37.037+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:37.037+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:37.037+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:37.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:37.037+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:37.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:37.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:37.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:06.165425+0000 (oldest deadline 2024-03-24T11:40:26.165425+0000) 2024-03-24T11:50:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:37.497+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:38.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:38.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:38.144+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:38.144+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:38.087+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:38.087+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:38.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:38.087+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:38.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:38.087+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:38.822 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:38.696+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29176 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:38.822 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:38.475+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:39.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:39.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:39.162+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:39.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:39.162+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:39.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:39.134+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:39.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:39.134+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:39.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:39.134+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:39.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:39.134+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:39.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:39.503+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:40.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:40.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:40.196+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:40.196+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:40.086+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:40.086+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:40.086+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:40.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:40.086+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:40.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:40.485+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:41.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:41.233+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:41.233+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:41.233+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:41.233+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:41.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:41.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:41.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:41.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:41.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:41.090+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:09.915740+0000 (oldest deadline 2024-03-24T11:40:29.915740+0000) 2024-03-24T11:50:41.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:41.090+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:41.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:41.440+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:42.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:42.133+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:42.280+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:42.280+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:42.280+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:42.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:42.281+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:42.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:42.490+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:43.243 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:43.120+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:43.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:43.468+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:43.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:43.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:43.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:43.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:43.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:43.242+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:40:13.879121+0000 (oldest deadline 2024-03-24T11:40:33.879121+0000) 2024-03-24T11:50:43.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:43.243+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:44.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:43.701+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29259 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:44.445 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:44.197+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:44.446 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:44.154+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:44.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:44.445+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:45.493 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:45.193+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:45.493 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:45.180+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:45.492+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:46.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:46.184+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:46.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:46.219+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:46.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:46.529+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:47.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:47.202+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:47.539 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:47.266+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:47.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:47.538+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:48.575 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:48.242+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:48.576 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:48.310+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:48.829 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:48.706+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29343 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:48.830 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:48.575+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:49.621 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:49.356+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:49.621 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:49.268+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:49.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:49.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:50.587 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:50.312+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:50.587 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:50.400+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:50.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:50.586+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:51.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:51.337+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:51.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:51.550+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:51.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:51.419+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:52.291+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:52.466+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:52.568+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:53.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:53.292+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:53.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:53.436+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:53.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:53.711+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29427 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:53.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:54.553 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:54.272+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:54.553 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:54.462+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:54.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:54.552+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:55.451 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:55.322+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:55.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:55.591+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:55.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:55.450+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:56.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:56.330+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:56.599 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:56.414+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:56.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:56.598+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:57.616 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:57.287+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:57.616 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:57.421+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:57.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:57.616+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:58.607 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:58.292+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:58.607 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:58.466+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:58.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:50:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:50:58.715+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29512 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:50:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:58.606+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:59.502 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:50:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:50:59.248+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:50:59.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:50:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:50:59.557+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:50:59.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:50:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:50:59.501+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:00.566 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:00.463+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:00.567 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:00.277+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:00.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:00.566+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:01.583 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:01.283+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:01.584 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:01.512+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:01.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:01.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:01.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:01.583+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:01.583+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:02.599 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:02.308+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:02.600 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:02.516+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:02.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:02.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:02.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:02.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:03.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:03.356+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:03.538+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:03.538+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:03.538+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:03.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:03.538+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:04.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:03.720+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29596 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:03.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:03.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:03.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:03.643+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:04.635 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:04.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:04.635 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:04.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:04.636 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:04.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:04.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:04.336+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:04.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:04.336+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:04.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:04.336+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:04.636 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:04.336+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:04.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:04.534+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:04.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:04.534+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:04.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:04.534+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:04.637 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:04.534+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:04.635+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:05.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:05.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:05.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:05.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:05.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:05.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:05.639 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:05.301+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:05.640 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:05.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:05.640 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:05.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:05.640 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:05.486+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:05.640 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:05.486+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:05.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:05.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:05.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:05.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:05.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:05.639+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:06.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:06.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:06.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:06.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:06.626 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:06.300+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:06.627 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:06.301+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:06.627 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:06.455+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:06.627 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:06.455+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:06.627 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:06.455+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:06.627 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:06.455+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:06.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:06.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:06.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:06.626+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:06.626+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:07.618 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:07.287+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:07.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:07.287+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:07.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:07.287+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:07.619 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:07.287+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:07.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:07.431+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:07.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:07.431+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:07.619 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:07.431+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:07.620 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:07.431+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:07.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:07.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:07.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:07.618+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:08.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:08.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:08.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:08.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:08.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:08.279+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:08.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:08.279+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:08.606 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:08.471+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:08.606 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:08.471+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:08.606 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:08.471+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:08.606 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:08.471+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:08.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:08.725+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29680 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:08.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:08.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:08.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:08.605+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:09.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:09.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:09.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:09.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:09.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:09.303+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:09.581 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:09.303+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:09.582 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:09.496+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:09.582 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:09.496+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:09.582 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:09.496+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:09.582 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:09.496+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:09.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:09.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:09.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:09.580+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:09.580+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:10.538 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:10.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:10.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:10.467+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:10.467+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:10.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:10.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:10.352+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:10.539 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:10.352+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:10.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:10.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:10.538+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:10.538+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:11.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:11.327+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:11.327+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:11.327+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:11.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:11.327+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:11.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:11.508+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:11.508+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:11.508+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:11.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:11.508+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:11.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:11.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:11.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:11.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:11.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:11.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:11.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:11.562+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:12.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:12.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:12.343+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:12.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:12.343+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:12.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:12.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:12.512+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:12.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:12.512+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:12.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:12.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:12.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:12.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:12.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:12.533+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:12.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:12.533+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:13.632 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:13.491+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:13.633 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:13.491+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:13.633 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:13.491+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:13.633 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:13.491+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:13.633 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:13.344+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:13.344+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:13.344+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:13.344+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:13.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:13.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:13.488+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:13.634 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:13.488+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:13.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:13.730+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29764 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:14.342+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:14.342+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:14.342+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:14.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:14.342+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:14.531+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:14.531+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:14.531+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:14.531+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:14.456+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:14.456+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:14.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:14.456+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:14.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:14.457+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:15.373+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:15.373+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:15.373+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:15.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:15.373+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:15.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:15.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:15.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:15.573+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:15.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:15.573+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:15.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:15.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:15.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:15.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:15.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:15.501+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:15.645 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:15.501+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:16.706 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:16.566+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:16.706 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:16.566+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:16.706 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:16.566+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:16.706 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:16.566+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:16.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:16.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:16.536+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:16.537+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:16.411+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:16.411+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:16.707 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:16.411+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:16.708 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:16.411+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:17.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:17.372+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:17.372+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:17.372+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:17.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:17.373+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:17.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:17.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:17.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:17.509+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:17.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:17.509+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:17.581+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:17.581+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:17.581+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:17.644 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:17.581+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:18.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:18.614+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:18.614+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:18.614+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:18.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:18.615+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:18.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:18.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:18.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:18.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:18.522+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:18.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:18.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:18.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:18.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:18.381+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:19.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:18.733+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29849 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:19.563+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:19.563+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:19.563+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:19.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:19.563+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:19.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:19.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:19.347+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:19.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:19.347+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:19.651+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:19.651+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:19.651+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:19.651+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:20.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:20.329+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:20.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:20.329+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:20.586 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:20.329+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:20.587 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:20.329+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:20.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:20.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:20.585+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:20.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:20.585+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:20.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:20.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:20.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:20.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:20.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:20.690+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:21.573 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:21.296+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:21.573 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:21.296+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:21.573 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:21.296+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:21.574 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:21.296+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:21.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:21.572+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:21.572+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:21.572+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:21.573+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:21.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:21.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:21.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:21.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:21.737+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:22.601 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:22.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:22.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:22.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:22.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:22.301+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:22.602 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:22.301+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:22.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:22.601+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:22.601+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:22.601+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:22.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:22.602+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:22.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:22.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:22.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:22.768+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:22.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:22.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:23.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:23.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:23.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:23.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:23.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:23.309+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:23.579 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:23.309+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:23.865 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:23.739+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 29933 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:23.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:23.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:23.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:23.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:23.865 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:23.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:23.866 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:23.579+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:23.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:23.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:23.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:23.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:23.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:23.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:23.866 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:23.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:24.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:24.276+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:24.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:24.276+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:24.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:24.276+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:24.604 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:24.277+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:24.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:24.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:24.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:24.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:24.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:24.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:24.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:24.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:24.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:24.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:24.763+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:25.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:25.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:25.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:25.250+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:25.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:25.251+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:25.920 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:25.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:25.920 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:25.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:25.920 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:25.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:25.920 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:25.731+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:25.920 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:25.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:25.921 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:25.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:25.921 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:25.650+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:25.921 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:25.650+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:26.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:26.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:26.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:26.206+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:26.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:26.207+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:27.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:26.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:26.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:26.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:26.677+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:26.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:26.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:26.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:26.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:27.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:27.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:27.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:27.255+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:27.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:27.256+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:28.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:27.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:27.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:27.668+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:27.669+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:27.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:27.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:28.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:27.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:27.736+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:28.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:28.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:28.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:28.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:28.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:28.286+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:28.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:28.287+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:28.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:28.744+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30018 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:28.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:28.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:28.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:28.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:28.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:28.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:28.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:28.716+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:28.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:28.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:28.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:28.743+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:28.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:28.743+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:29.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:29.331+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:29.331+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:29.331+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:29.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:29.332+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:30.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:29.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:29.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:29.678+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:30.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:29.678+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:29.710+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:29.710+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:30.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:29.710+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:30.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:29.710+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:30.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:30.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:30.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:30.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:30.354+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:31.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:30.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:30.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:30.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:30.695+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:30.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:30.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:31.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:30.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:31.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:30.707+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:31.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:31.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:31.366+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:31.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:31.366+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:32.064 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:31.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:32.064 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:31.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:32.064 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:31.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:32.064 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:31.678+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:32.064 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:31.720+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:32.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:31.720+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:32.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:31.720+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:32.065 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:31.720+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:32.710 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:32.410+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:32.710 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:32.410+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:32.710 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:32.410+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:32.710 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:32.410+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:33.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:32.709+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:32.709+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:32.709+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:32.710+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:32.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:32.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:32.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:32.752+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:33.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:33.390+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:33.390+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:33.390+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:33.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:33.391+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:34.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:33.749+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30102 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:33.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:33.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:33.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:33.686+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:34.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:33.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:33.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:33.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:33.777+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:34.706 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:34.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:34.707 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:34.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:34.707 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:34.423+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:34.707 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:34.423+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:35.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:34.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:35.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:34.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:35.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:34.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:35.136 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:34.823+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:35.137 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:34.706+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:35.137 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:34.706+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:35.137 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:34.706+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:35.137 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:34.706+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:35.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:35.379+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:35.379+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:35.379+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:35.379+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:36.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:35.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:35.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:35.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:35.675+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:35.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:35.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:35.840+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:35.841+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:36.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:36.335+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:36.335+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:36.335+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:36.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:36.335+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:36.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:36.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:36.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:36.676+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:36.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:36.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:36.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:37.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:36.794+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:37.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:37.358+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:37.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:37.358+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:37.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:37.358+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:37.629 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:37.358+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:37.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:37.628+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:37.628+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:37.628+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:37.629+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:37.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:37.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:37.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:37.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:37.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:37.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:38.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:38.405+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:38.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:38.405+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:38.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:38.405+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:38.726 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:38.405+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:38.727 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:38.630+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:38.727 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:38.630+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:38.727 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:38.630+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:38.727 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:38.630+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:38.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:38.754+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30187 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:38.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:38.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:38.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:38.725+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:39.758 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:39.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:39.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:39.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:39.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:39.401+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:39.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:39.402+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:39.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:39.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:39.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:39.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:39.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:39.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:39.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:39.625+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:40.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:39.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:39.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:39.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:39.758+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:40.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:40.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:40.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:40.435+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:40.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:40.435+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:40.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:40.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:40.717+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:40.718+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:40.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:40.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:40.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:41.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:40.660+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:41.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:41.419+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:41.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:41.419+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:41.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:41.420+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:41.638 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:41.420+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:41.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:41.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:41.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:41.638+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:41.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:41.638+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:41.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:41.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:41.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:41.746+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:42.781 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:42.612+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:42.781 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:42.612+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:42.782 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:42.612+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:42.782 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:42.612+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:42.782 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:42.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:42.782 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:42.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:42.782 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:42.413+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:42.783 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:42.413+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:43.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:42.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:42.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:42.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:42.781+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:43.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:43.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:43.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:43.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:43.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:43.463+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:43.759 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:43.463+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:43.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:43.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:43.759 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:43.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:43.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:43.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:43.760 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:43.605+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:44.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:43.759+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30274 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:43.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:43.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:43.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:44.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:43.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:44.605 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:44.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:44.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:44.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:44.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:44.427+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:44.606 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:44.427+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:44.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:44.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:44.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:44.802+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:44.802+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:44.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:44.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:44.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:44.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:44.606+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:45.389+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:45.389+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:45.389+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:45.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:45.389+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:45.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:45.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:45.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:45.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:45.605+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:45.644 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:45.605+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:45.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:45.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:45.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:45.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:46.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:46.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:46.354+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:46.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:46.354+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:47.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:46.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:46.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:46.646+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:46.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:46.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:46.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:47.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:46.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:46.712+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:47.656 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:47.396+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:47.657 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:47.396+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:47.657 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:47.396+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:47.657 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:47.396+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:48.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:47.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:47.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:47.657+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:47.657+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:47.691+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:47.691+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:48.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:47.691+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:47.691+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:48.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:48.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:48.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:48.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:48.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:48.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:48.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:48.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:48.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:48.381+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:48.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:48.382+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:48.978 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:48.763+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30358 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:48.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:48.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:48.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:48.682+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:49.602+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:49.602+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:49.602+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:49.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:49.602+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:49.643 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:49.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:49.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:49.388+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:49.644 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:49.388+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:50.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:49.694+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:49.694+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:49.694+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:50.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:49.694+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:50.495 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:50.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:50.496 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:50.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:50.496 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:50.407+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:50.496 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:50.407+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:50.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:50.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:50.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:50.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:50.633+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:50.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:50.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:50.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:50.690+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:50.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:50.690+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:51.680 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:51.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:51.680 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:51.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:51.680 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:51.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:51.681 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:51.426+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:52.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:51.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:51.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:51.679+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:51.680+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:51.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:51.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:51.682+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:51.683+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:52.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:52.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:52.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:52.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:52.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:52.425+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:52.729 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:52.425+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:52.729 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:52.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:52.730 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:52.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:52.730 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:52.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:52.730 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:52.657+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:53.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:52.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:52.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:52.729+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:52.729+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:53.756 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:53.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:53.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:53.442+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:53.443+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:53.609+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:53.609+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:53.609+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:53.757 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:53.609+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:54.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:53.768+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30442 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:53.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:53.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:53.756+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:54.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:53.756+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:54.747 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:54.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:54.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:54.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:54.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:54.464+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:54.748 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:54.464+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:54.748 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:54.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:54.748 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:54.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:54.749 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:54.652+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:54.749 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:54.652+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:55.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:54.747+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:55.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:54.747+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:54.747+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:54.748+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:55.769 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:55.500+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:55.769 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:55.500+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:55.770 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:55.500+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:55.770 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:55.500+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:55.770 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:55.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:55.770 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:55.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:55.770 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:55.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:55.770 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:55.671+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:56.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:55.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:55.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:55.769+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:55.769+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:56.768 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:56.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:56.768 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:56.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:56.768 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:56.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:56.768 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:56.516+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:56.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:56.662+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:56.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:56.662+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:56.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:56.662+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:56.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:56.662+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:57.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:56.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:56.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:56.766+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:57.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:56.767+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:57.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:57.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:57.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:57.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:57.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:57.512+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:57.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:57.653+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:57.653+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:57.653+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:57.653+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:57.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:57.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:57.725+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:57.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:57.725+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:58.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:51:58.772+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30526 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:51:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:58.687+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:58.687+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:58.687+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:58.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:58.687+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:58.555+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:58.555+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:58.555+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:58.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:58.555+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:58.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:58.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:58.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:58.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:58.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:58.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:58.754+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:59.559+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:59.559+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:59.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:59.559+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:51:59.895 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:51:59.559+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:51:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:59.696+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:59.696+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:59.696+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:51:59.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:51:59.697+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:51:59.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:59.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:59.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:59.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:59.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:59.754+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:51:59.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:51:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:51:59.754+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:00.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:00.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:00.763+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:00.763+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:00.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:00.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:00.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:00.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:00.516+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:00.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:00.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:00.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:00.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:00.730+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:00.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:00.730+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:01.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:01.740+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:01.740+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:01.740+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:01.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:01.740+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:01.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:01.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:01.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:01.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:01.542+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:01.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:01.542+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:01.710+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:01.710+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:01.710+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:01.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:01.710+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:02.788+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:02.788+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:02.788+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:02.788+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:02.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:02.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:02.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:02.573+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:02.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:02.574+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:02.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:02.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:02.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:02.686+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:03.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:03.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:03.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:03.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:03.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:03.526+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:03.779 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:03.527+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:03.780 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:03.661+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:03.780 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:03.661+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:03.780 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:03.661+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:03.780 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:03.662+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:04.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:03.778+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30610 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:03.822+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:03.822+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:03.822+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:04.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:03.822+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:04.854 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:04.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:04.855 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:04.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:04.855 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:04.529+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:04.855 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:04.529+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:04.855 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:04.640+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:04.855 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:04.640+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:04.856 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:04.640+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:04.856 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:04.640+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:05.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:04.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:04.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:04.854+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:04.855+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:05.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:05.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:05.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:05.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:05.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:05.523+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:05.824 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:05.523+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:05.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:05.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:05.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:05.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:05.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:05.657+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:05.825 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:05.657+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:06.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:05.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:06.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:05.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:05.823+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:05.824+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:06.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:06.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:06.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:06.782+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:06.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:06.564+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:06.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:06.564+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:06.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:06.564+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:06.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:06.564+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:06.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:06.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:06.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:06.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:06.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:06.622+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:06.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:06.623+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:07.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:07.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:07.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:07.577+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:07.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:07.578+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:07.771+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:07.771+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:07.771+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:07.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:07.771+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:07.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:07.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:07.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:07.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:07.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:07.593+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:07.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:07.593+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:08.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:08.783+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30694 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:08.772+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:08.772+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:08.772+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:08.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:08.773+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:08.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:08.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:08.613+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:08.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:08.614+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:08.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:08.591+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:08.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:08.591+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:08.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:08.591+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:08.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:08.592+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:09.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:09.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:09.733+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:09.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:09.734+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:09.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:09.660+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:09.660+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:09.660+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:09.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:09.660+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:09.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:09.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:09.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:09.596+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:09.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:09.596+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:10.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:10.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:10.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:10.757+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:10.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:10.757+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:10.691+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:10.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:10.691+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:10.691+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:10.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:10.691+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:10.601+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:10.601+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:10.601+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:10.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:10.601+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:11.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:11.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:11.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:11.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:11.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:11.787+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:11.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:11.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:11.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:11.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:11.695+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:11.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:11.696+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:11.605+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:11.605+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:11.605+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:11.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:11.605+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:12.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:12.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:12.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:12.827+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:12.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:12.827+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:12.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:12.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:12.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:12.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:12.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:12.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:12.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:12.694+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:12.633+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:12.633+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:12.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:12.633+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:12.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:12.633+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:13.913 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:13.788+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30778 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:13.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:13.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:13.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:13.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:13.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:13.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:13.914 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:13.699+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:13.914 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:13.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:13.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:13.786+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:13.787+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:13.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:13.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:13.668+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:13.915 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:13.668+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:15.071 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:14.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:15.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:14.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:15.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:14.806+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:15.072 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:14.807+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:15.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:14.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:15.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:14.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:15.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:14.656+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:15.073 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:14.657+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:15.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:14.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:15.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:14.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:15.073 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:14.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:15.074 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:14.716+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:16.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:15.665+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:15.665+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:15.665+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:15.665+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:15.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:15.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:15.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:15.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:15.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:15.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:16.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:15.825+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:16.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:15.825+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:17.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:16.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:16.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:16.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:16.694+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:16.866+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:16.866+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:16.866+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:16.867+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:16.749+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:16.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:16.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:16.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:17.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:17.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:17.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:18.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:17.840+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:18.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:17.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:17.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:17.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:18.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:17.701+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:18.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:17.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:18.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:17.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:18.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:17.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:18.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:17.753+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:18.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:18.793+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30862 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:18.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:18.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:18.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:18.694+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:18.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:18.695+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:18.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:18.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:18.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:18.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:18.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:18.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:18.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:18.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:18.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:18.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:18.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:18.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:18.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:18.813+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:20.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:19.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:19.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:19.664+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:19.665+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:19.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:19.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:19.815+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:19.816+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:19.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:19.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:19.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:19.781+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:21.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:20.784+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:20.784+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:20.784+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:21.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:20.784+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:20.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:20.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:20.811+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:21.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:20.812+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:20.697+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:20.697+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:20.697+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:21.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:20.697+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:22.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:21.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:21.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:21.811+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:21.812+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:21.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:21.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:21.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:21.780+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:22.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:21.707+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:22.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:21.707+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:22.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:21.707+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:22.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:21.707+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:23.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:22.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:22.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:22.839+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:22.839+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:22.726+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:22.726+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:22.726+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:22.726+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:22.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:22.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:22.781+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:22.781+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:23.926 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:23.798+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 30946 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:23.927 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:23.810+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:23.927 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:23.810+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:23.927 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:23.810+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:23.927 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:23.810+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:23.928 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:23.709+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:23.928 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:23.709+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:23.928 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:23.709+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:23.928 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:23.709+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:23.928 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:23.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:23.928 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:23.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:23.929 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:23.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:23.929 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:23.735+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:24.820+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:24.820+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:24.820+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:24.820+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:25.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:24.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:24.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:24.700+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:24.700+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:25.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:24.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:25.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:24.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:25.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:24.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:25.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:24.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:26.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:25.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:25.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:25.680+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:25.680+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:25.844+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:25.844+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:25.844+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:26.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:25.844+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:25.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:25.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:25.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:26.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:25.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:27.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:26.646+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:26.646+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:26.646+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:26.646+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:26.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:26.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:26.737+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:26.737+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:27.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:26.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:27.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:26.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:27.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:26.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:27.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:26.849+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:27.891 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:27.598+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:27.891 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:27.598+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:27.891 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:27.598+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:27.891 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:27.598+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:27.891 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:27.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:27.891 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:27.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:27.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:27.741+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:27.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:27.741+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:28.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:27.890+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:27.890+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:27.890+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:27.890+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:28.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:28.804+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31030 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:28.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:28.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:28.697+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:28.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:28.697+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:28.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:28.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:28.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:28.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:28.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:28.588+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:28.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:28.588+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:29.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:28.915+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:28.915+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:28.915+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:28.915+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:29.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:29.603+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:29.603+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:29.603+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:29.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:29.603+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:29.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:29.709+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:29.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:29.709+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:29.709+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:29.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:29.709+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:30.231 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:29.901+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:30.231 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:29.901+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:30.231 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:29.901+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:30.231 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:29.901+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:30.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:30.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:30.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:30.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:30.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:30.556+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:30.880 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:30.557+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:30.880 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:30.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:30.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:30.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:30.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:30.751+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:30.881 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:30.751+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:31.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:30.880+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:30.880+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:30.880+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:31.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:30.880+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:31.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:31.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:31.768 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:31.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:31.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:31.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:31.769 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:31.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:31.769 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:31.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:31.769 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:31.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:31.769 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:31.512+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:31.769 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:31.512+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:32.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:31.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:31.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:31.859+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:32.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:31.859+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:32.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:32.504+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:32.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:32.504+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:32.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:32.504+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:32.772 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:32.504+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:33.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:32.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:32.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:32.907+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:33.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:32.908+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:32.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:32.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:32.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:32.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:33.810 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:33.531+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:33.810 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:33.531+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:33.810 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:33.531+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:33.810 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:33.531+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:34.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:33.809+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31114 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:33.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:33.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:34.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:33.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:34.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:33.905+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:33.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:33.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:33.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:34.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:33.814+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:34.813 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:34.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:34.813 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:34.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:34.813 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:34.516+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:34.814 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:34.516+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:35.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:34.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:34.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:34.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:34.813+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:34.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:34.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:35.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:34.897+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:35.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:34.897+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:35.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:35.489+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:35.798 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:35.489+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:35.799 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:35.489+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:35.799 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:35.489+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:36.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:35.942+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:35.942+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:35.942+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:36.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:35.942+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:35.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:35.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:36.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:35.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:36.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:35.799+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:36.770 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:36.476+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:36.770 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:36.476+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:36.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:36.476+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:36.771 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:36.476+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:37.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:36.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:36.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:36.769+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:36.769+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:36.982+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:36.982+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:36.982+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:37.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:36.982+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:37.780 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:37.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:37.780 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:37.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:37.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:37.474+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:37.781 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:37.474+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:38.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.023+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.023+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.023+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.023+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:37.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:37.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:37.779+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:38.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:37.779+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:38.816 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:38.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:38.816 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:38.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:38.816 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:38.487+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:38.816 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:38.487+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:39.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:38.814+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31195 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.975+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.975+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.975+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:39.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:38.975+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:38.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:38.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:38.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:38.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:39.842 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:39.491+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:39.843 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:39.491+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:39.843 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:39.491+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:39.843 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:39.492+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:40.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:39.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:39.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:39.841+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:40.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:39.841+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:39.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:40.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:39.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:40.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:39.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:40.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:39.944+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:40.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:40.538+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:40.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:40.538+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:40.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:40.538+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:40.837 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:40.538+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:41.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:40.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:41.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:40.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:40.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:41.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:40.836+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:40.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:40.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:40.988+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:41.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:40.988+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:41.552+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:41.552+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:41.552+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:41.552+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:41.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:41.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:41.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:41.796+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:41.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:41.796+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:42.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:41.945+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:41.945+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:41.945+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:41.946+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:42.808 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:42.553+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:42.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:42.553+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:42.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:42.553+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:42.809 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:42.553+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:43.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:42.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:42.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:42.808+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:43.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:42.808+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:42.980+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:43.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:42.980+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:42.980+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:43.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:42.981+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:43.846 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:43.820+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31270 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:43.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:43.592+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:43.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:43.592+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:43.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:43.592+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:43.847 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:43.593+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:43.847 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:43.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:43.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:43.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:43.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:43.761+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:43.848 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:43.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:44.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:43.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:44.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:43.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:44.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:43.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:44.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:43.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:44.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:44.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:44.640+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:44.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:44.640+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:44.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:44.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:44.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:44.761+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:45.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:44.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:44.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:44.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:44.928+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:45.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:45.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:45.905+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:46.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:45.906+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:45.679+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:45.679+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:45.679+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:46.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:45.680+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:45.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:45.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:45.758+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:46.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:45.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:47.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:46.895+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:46.895+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:46.895+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:47.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:46.896+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:47.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:46.718+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:46.718+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:46.718+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:46.719+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:46.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:46.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:46.719+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:47.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:46.719+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:48.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:47.889+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:47.889+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:47.889+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:48.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:47.889+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:48.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:47.732+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:47.732+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:47.732+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:48.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:47.732+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:47.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:47.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:48.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:47.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:48.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:47.690+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:48.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:48.825+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31344 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:48.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:48.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:48.711+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:48.712+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:48.682+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:48.682+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:48.682+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:48.894 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:48.683+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:49.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:48.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:48.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:48.927+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:48.927+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:49.685+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:49.685+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:49.685+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:50.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:49.685+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:49.685+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:49.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:49.686+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:50.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:49.686+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:49.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:50.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:49.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:50.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:49.911+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:50.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:49.912+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:51.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:50.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:50.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:50.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:51.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:50.671+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:50.867+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:50.867+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:51.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:50.867+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:51.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:50.868+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:50.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:50.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:50.723+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:51.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:50.723+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:52.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:51.835+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:51.835+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:51.835+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:51.836+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:51.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:51.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:52.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:51.744+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:51.745+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:51.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:51.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:51.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:51.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:53.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:52.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:52.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:52.787+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:52.787+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:52.674+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:53.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:52.674+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:52.674+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:52.675+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:53.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:52.813+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:53.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:52.813+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:53.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:52.813+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:53.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:52.814+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:53.956 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:53.830+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31419 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:53.956 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:53.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:53.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:53.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:53.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:53.831+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:53.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:53.831+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:53.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:53.662+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:53.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:53.662+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:53.957 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:53.662+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:53.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:53.662+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:53.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:53.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:53.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:53.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:53.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:53.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:53.958 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:53.754+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:55.007 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:54.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:55.008 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:54.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:55.008 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:54.739+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:55.008 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:54.739+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:55.008 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:54.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:55.008 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:54.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:55.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:54.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:55.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:54.658+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:55.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:54.870+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:55.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:54.870+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:55.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:54.870+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:55.009 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:54.870+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:55.903 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:55.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:55.904 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:55.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:55.904 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:55.784+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:55.904 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:55.785+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:55.904 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:55.642+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:55.904 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:55.642+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:55.904 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:55.642+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:55.905 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:55.642+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:56.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:55.903+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:56.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:55.903+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:55.903+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:56.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:55.903+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:56.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:56.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:56.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:56.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:56.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:56.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:56.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:56.819+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:56.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:56.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:56.621+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:56.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:56.621+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:57.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:56.933+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:57.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:56.933+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:56.933+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:57.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:56.933+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:57.965 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:57.866+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:57.965 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:57.866+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:57.966 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:57.866+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:57.966 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:57.866+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:57.966 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:57.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:57.966 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:57.671+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:57.966 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:57.672+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:57.966 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:57.672+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:58.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:57.965+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:57.965+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:57.965+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:58.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:57.965+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:58.960 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:52:58.836+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31493 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:52:58.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:58.931+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:58.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:58.931+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:58.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:58.931+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:52:58.961 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:58.931+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:52:58.962 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:58.858+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:58.962 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:58.858+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:58.962 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:58.858+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:52:58.962 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:58.859+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:52:58.962 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:58.704+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:58.963 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:58.704+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:58.963 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:58.704+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:52:58.963 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:58.705+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:00.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:59.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:59.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:59.736+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:00.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:52:59.737+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:59.882+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:59.882+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:00.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:59.882+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:00.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:52:59.882+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:59.897+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:59.897+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:59.897+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:52:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:52:59.898+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:01.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:00.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:00.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:00.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:01.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:00.689+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:00.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:00.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:01.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:00.928+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:01.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:00.928+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:00.887+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:00.887+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:00.887+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:01.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:00.887+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:02.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:01.903+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:01.903+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:01.903+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:01.904+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:01.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:01.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:01.706+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:01.706+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:02.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:01.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:02.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:01.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:02.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:01.948+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:02.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:01.948+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:02.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:02.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:02.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:03.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:02.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:02.925+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:02.925+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:02.925+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:02.925+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:02.922+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:02.922+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:03.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:02.922+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:03.145 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:02.923+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:03.916 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:03.841+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31574 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:03.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:03.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:03.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:03.763+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:03.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:03.764+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:03.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:03.764+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:04.223 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:03.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:04.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:03.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:04.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:03.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:04.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:03.916+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:04.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:03.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:04.224 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:03.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:04.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:03.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:04.225 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:03.961+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:05.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:04.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:04.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:04.812+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:05.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:04.812+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:04.975+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:04.975+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:04.975+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:04.975+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:04.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:04.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:04.926+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:05.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:04.926+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:06.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:05.968+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:05.968+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:05.968+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:05.969+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:05.950+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:05.950+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:05.950+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:05.950+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:06.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:05.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:05.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:05.799+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:05.799+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:07.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:06.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:06.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:06.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:06.970+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:06.951+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:06.951+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:06.951+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:07.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:06.951+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:06.821+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:06.821+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:06.821+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:06.821+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:07.966+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:07.966+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:07.966+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:07.966+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:07.991+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:07.991+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:07.991+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:07.991+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:07.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:07.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:08.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:07.831+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:08.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:07.831+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:09.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:08.846+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31657 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:08.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:08.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:08.960+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:09.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:08.960+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:09.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:08.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:09.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:08.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:09.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:08.961+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:09.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:08.961+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:09.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:08.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:09.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:08.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:09.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:08.794+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:09.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:08.794+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:10.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:09.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:09.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:09.949+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:09.949+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:10.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:09.929+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:10.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:09.929+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:09.929+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:09.929+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:09.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:09.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:09.748+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:09.748+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:10.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:10.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:10.747+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:11.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:10.747+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:10.995+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:10.995+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:10.995+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:10.995+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:11.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:10.887+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:11.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:10.887+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:11.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:10.887+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:11.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:10.887+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:12.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:11.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:11.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:11.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:11.959+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:11.858+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:11.858+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:11.858+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:11.858+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:11.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:11.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:11.745+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:11.745+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:13.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:12.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:12.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:12.944+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:12.944+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:12.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:12.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:12.815+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:12.815+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:12.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:12.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:12.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:12.774+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:14.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:13.851+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31741 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:13.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:13.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:13.809+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:14.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:13.809+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:13.964+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:13.964+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:13.964+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:13.965+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:13.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:13.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:13.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:14.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:13.812+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:15.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:14.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:14.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:14.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:14.780+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:14.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:14.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:14.958+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:15.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:14.958+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:14.789+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:14.789+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:14.789+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:14.789+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:16.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:15.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:16.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:15.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:15.746+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:15.746+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:15.770+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:15.770+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:15.770+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:16.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:15.770+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:15.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:15.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:15.984+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:16.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:15.984+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:16.803 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:16.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:16.803 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:16.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:16.803 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:16.727+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:16.803 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:16.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:17.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:17.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:17.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:17.013+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:17.013+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:16.802+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:16.802+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:16.802+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:16.802+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:18.003 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:17.773+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:18.004 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:17.773+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:18.004 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:17.773+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:18.004 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:17.773+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:18.004 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:17.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:18.004 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:17.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:18.005 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:17.728+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:18.005 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:17.728+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:18.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:18.002+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:18.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:18.002+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:18.002+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:18.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:18.002+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:19.030 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:18.855+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31823 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:19.030 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:18.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:19.030 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:18.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:19.031 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:18.772+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:19.031 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:18.772+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:19.031 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:18.771+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:19.031 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:18.771+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:19.031 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:18.771+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:19.032 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:18.771+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:19.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:19.028+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:19.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:19.028+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:19.028+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:19.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:19.029+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:20.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:19.819+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:20.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:19.819+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:20.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:19.819+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:20.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:19.819+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:20.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:19.740+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:20.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:19.740+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:20.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:19.740+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:20.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:19.740+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:20.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:20.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:20.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:20.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:20.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:20.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:20.026+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:21.047 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:20.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:21.047 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:20.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:21.047 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:20.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:21.047 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:20.775+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:21.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:20.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:21.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:20.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:21.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:20.735+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:21.048 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:20.735+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:21.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:21.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:21.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:21.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:21.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:21.046+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:21.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:21.046+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:22.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:21.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:22.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:21.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:22.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:21.812+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:22.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:21.812+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:22.022 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:21.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:22.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:21.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:22.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:21.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:22.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:21.707+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:22.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:22.021+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:22.021+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:22.021+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:22.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:22.022+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:23.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:22.789+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:23.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:22.789+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:23.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:22.789+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:23.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:22.789+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:23.060 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:22.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:23.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:22.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:23.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:22.658+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:23.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:22.659+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:23.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:23.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:23.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:23.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:23.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:23.060+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:23.988 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:23.861+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31907 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:23.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:23.833+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:23.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:23.833+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:23.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:23.833+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:23.989 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:23.834+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:23.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:23.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:23.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:23.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:23.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:23.689+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:23.990 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:23.689+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:24.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:24.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:24.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:24.086+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:24.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:24.087+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:25.100 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:24.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:25.101 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:24.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:25.101 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:24.722+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:25.101 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:24.722+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:25.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:24.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:25.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:24.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:25.101 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:24.820+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:25.102 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:24.820+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:25.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:25.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:25.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:25.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:25.100+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:25.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:25.101+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:26.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:25.860+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:26.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:25.860+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:26.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:25.860+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:26.060 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:25.861+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:26.060 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:25.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:26.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:25.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:26.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:25.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:26.061 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:25.751+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:26.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:26.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:26.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:26.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:26.059+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:26.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:26.060+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:27.051 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:26.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:27.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:26.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:27.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:26.774+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:27.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:26.774+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:27.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:26.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:27.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:26.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:27.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:26.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:27.053 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:26.880+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:27.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:27.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:27.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:27.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:27.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:27.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:27.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:27.052+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:28.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:27.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:27.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:27.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:28.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:27.917+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:28.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:28.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:28.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:28.045+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:28.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:28.046+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:27.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:27.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:27.813+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:28.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:27.814+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:29.026 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:28.867+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 31989 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:28.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:28.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:28.766+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:28.766+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:28.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:28.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:29.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:28.900+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:29.028 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:28.901+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:29.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:29.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:29.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:29.026+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:29.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:29.026+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:30.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:29.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:30.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:29.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:30.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:29.760+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:30.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:29.760+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:30.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:29.863+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:30.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:29.863+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:30.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:29.863+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:30.072 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:29.863+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:30.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:30.070+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:30.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:30.070+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:30.070+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:30.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:30.070+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:31.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:30.914+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:31.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:30.914+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:31.104 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:30.914+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:31.105 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:30.914+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:31.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:30.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:31.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:30.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:31.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:30.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:31.105 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:30.794+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:31.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:31.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:31.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:31.104+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:31.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:31.104+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:32.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:31.910+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:32.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:31.910+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:32.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:31.910+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:32.111 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:31.911+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:32.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:31.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:32.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:31.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:32.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:31.783+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:32.112 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:31.784+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:32.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:32.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:32.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:32.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:32.110+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:32.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:32.111+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:33.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:32.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:32.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:32.901+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:33.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:32.901+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:32.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:32.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:33.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:32.765+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:33.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:32.765+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:33.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:33.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:33.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:33.144+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:33.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:33.144+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:33.995 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:33.872+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32072 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:33.996 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:33.894+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:33.996 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:33.894+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:33.996 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:33.894+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:33.996 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:33.894+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:33.997 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:33.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:33.997 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:33.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:33.997 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:33.753+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:33.997 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:33.753+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:34.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:34.132+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:34.132+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:34.132+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:34.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:34.133+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:35.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:34.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:34.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:34.849+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:35.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:34.850+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:34.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:34.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:35.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:34.725+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:35.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:34.725+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:35.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:35.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:35.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:35.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:35.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:35.147+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:35.643 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:35.147+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:36.122 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:35.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:36.122 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:35.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:36.122 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:35.752+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:36.122 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:35.752+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:36.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:35.837+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:36.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:35.837+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:36.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:35.837+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:36.123 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:35.838+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:36.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:36.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:36.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:36.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:36.121+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:36.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:36.122+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:37.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:36.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:37.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:36.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:37.088 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:36.793+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:37.089 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:36.793+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:37.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:36.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:37.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:36.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:37.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:36.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:37.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:36.829+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:37.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:37.087+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:37.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:37.087+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:37.087+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:37.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:37.088+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:38.120 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:37.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:38.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:37.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:38.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:37.822+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:38.121 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:37.822+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:38.121 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:37.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:38.121 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:37.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:38.122 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:37.750+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:38.122 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:37.750+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:38.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:38.120+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:38.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:38.120+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:38.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:38.120+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:38.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:38.121+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:38.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:38.878+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32154 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:38.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:38.825+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:38.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:38.825+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:38.825+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:38.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:38.825+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:38.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:38.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:38.707+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:38.708+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:39.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:39.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:39.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:39.136+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:39.137+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:39.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:39.137+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:40.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:39.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:40.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:39.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:40.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:39.732+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:40.095 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:39.732+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:40.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:39.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:40.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:39.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:40.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:39.817+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:40.096 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:39.817+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:40.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:40.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:40.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:40.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:40.095+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:40.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:40.095+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:41.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:40.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:41.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:40.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:41.084 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:40.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:41.085 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:40.818+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:41.085 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:40.688+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:41.085 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:40.688+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:41.085 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:40.688+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:41.085 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:40.688+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:41.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:41.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:41.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:41.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:41.084+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:41.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:41.084+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:42.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:41.834+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:42.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:41.834+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:42.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:41.834+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:42.053 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:41.834+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:42.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:41.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:42.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:41.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:42.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:41.680+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:42.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:41.680+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:42.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:42.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:42.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:42.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:42.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:42.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:42.052+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:43.020 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:42.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:43.020 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:42.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:43.020 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:42.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:43.021 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:42.796+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:43.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:42.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:43.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:42.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:43.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:42.716+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:43.021 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:42.716+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:43.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:43.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:43.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:43.020+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:43.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:43.020+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:44.008 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:43.883+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32237 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:44.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:43.669+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:44.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:43.669+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:44.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:43.669+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:44.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:43.669+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:44.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:43.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:44.009 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:43.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:44.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:43.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:44.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:43.769+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:44.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:44.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:44.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:44.034+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:44.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:44.034+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:44.674 ERROR:paramiko.transport:Socket exception: No route to host (113) 2024-03-24T11:53:44.680 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-03-24T11:53:44.680 INFO:tasks.workunit:Stopping ['suites/fsstress.sh'] on client.1... 2024-03-24T11:53:44.681 ERROR:teuthology.run_tasks:Manager failed: kclient Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T11:53:44.694 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-03-24T11:53:44.708 INFO:tasks.cephadm:Teardown begin 2024-03-24T11:53:44.708 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T11:53:44.713 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-03-24T11:53:44.752 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:53:44.753 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:53:45.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:44.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:45.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:44.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:45.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:44.714+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:45.010 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:44.714+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:45.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:44.748+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:45.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:44.748+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:45.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:44.748+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:45.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:44.749+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:45.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:45.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:45.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:45.008+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:45.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:45.009+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:46.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:45.782+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:46.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:45.782+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:46.052 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:45.782+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:46.053 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:45.783+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:46.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:45.691+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:46.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:45.691+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:46.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:45.691+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:46.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:45.691+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:46.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:46.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:46.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:46.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:46.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:46.051+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:46.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:46.051+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:47.027 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:46.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:47.028 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:46.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:47.028 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:46.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:47.028 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:46.769+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:47.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:46.734+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:47.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:46.734+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:47.028 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:46.734+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:47.029 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:46.734+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:47.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:47.027+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:47.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:47.027+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:47.027+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:47.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:47.027+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:47.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:53:48.055 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:47.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:48.056 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:47.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:48.056 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:47.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:48.056 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:47.735+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:48.056 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:47.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:48.056 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:47.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:48.056 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:47.759+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:48.057 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:47.759+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:48.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:48.055+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:48.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:48.055+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:48.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:48.055+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:48.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:48.055+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:48.976 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:48.888+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32319 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:48.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:48.689+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:48.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:48.689+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:48.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:48.689+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:48.977 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:48.689+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:48.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:48.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:48.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:48.770+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:48.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:48.771+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:49.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:49.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:49.038+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:49.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:49.038+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:50.034 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:49.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:50.034 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:49.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:50.034 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:49.699+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:50.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:49.700+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:50.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:49.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:50.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:49.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:50.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:49.780+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:50.035 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:49.781+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:50.303 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:50.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:50.304 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:50.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:50.304 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:50.033+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:50.304 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:50.034+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:51.010 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:50.729+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:51.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:50.729+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:51.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:50.729+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:51.011 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:50.730+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:51.011 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:50.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:51.011 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:50.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:51.011 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:50.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:51.012 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:50.824+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:51.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.011+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:51.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.011+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.011+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:51.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.011+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:52.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.981+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:52.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:51.981+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:52.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:51.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:51.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:51.829+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:52.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:51.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:51.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:52.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:51.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:52.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:51.750+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:52.145 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:51.750+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:53.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:52.978+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:52.978+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:52.978+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:53.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:52.978+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:53.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:52.791+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:52.791+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:52.791+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:53.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:52.791+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:52.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:52.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:53.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:52.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:53.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:52.872+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:53.916 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:53.893+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32404 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:53.916 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:53.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:53.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:53.829+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:53.829+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:53.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:53.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:53.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:53.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:53.837+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:54.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:53.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:54.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:53.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:54.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:53.985+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:54.393 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:53.985+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:55.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:54.960+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:54.960+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:54.960+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:55.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:54.960+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:54.838+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:54.838+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:55.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:54.838+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:55.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:54.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:54.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:54.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:54.809+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:55.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:54.809+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:56.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:55.953+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:55.953+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:55.953+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:56.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:55.953+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:55.778+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:55.778+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:55.778+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:56.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:55.778+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:55.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:55.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:55.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:56.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:55.830+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:56.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:53:56.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:53:57.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:56.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:56.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:56.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:57.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:56.798+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:57.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:56.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:56.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:56.858+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:56.858+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:56.921+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:56.921+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:56.921+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:57.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:56.921+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:58.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:57.773+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:58.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:57.773+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:58.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:57.773+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:58.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:57.773+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:58.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:57.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:58.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:57.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:58.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:57.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:58.046 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:57.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:58.047 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:57.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:58.047 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:57.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:58.047 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:57.877+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:58.047 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:57.877+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:59.021 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:53:58.897+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32487 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:53:59.021 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:58.889+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:59.022 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:58.889+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:59.022 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:58.889+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:53:59.022 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:58.890+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:59.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:58.747+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:59.022 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:58.747+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:59.023 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:58.747+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:53:59.023 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:58.748+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:53:59.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:58.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:59.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:58.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:59.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:58.803+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:53:59.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:58.803+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:53:59.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:00.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:59.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:59.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:59.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:00.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:53:59.843+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:59.735+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:59.735+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:00.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:59.735+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:00.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:53:59.736+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:59.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:59.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:59.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:00.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:53:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:53:59.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:01.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:00.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:00.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:00.777+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:01.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:00.777+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:00.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:00.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:01.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:00.802+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:01.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:00.802+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:00.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:00.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:00.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:01.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:00.884+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:02.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:01.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:02.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:01.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:01.794+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:01.794+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:01.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:01.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:02.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:01.826+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:02.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:01.826+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:01.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:01.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:01.836+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:02.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:01.836+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:02.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:02.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:02.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:02.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:02.835+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:02.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:02.835+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:02.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:02.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:02.812+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:02.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:02.812+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:02.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:02.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:02.830+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:02.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:02.831+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:03.917 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:03.904+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32571 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:03.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:03.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:03.917 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:03.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:03.918 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:03.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:03.918 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:03.828+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:03.918 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:03.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:03.918 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:03.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:03.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:03.842+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:03.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:03.843+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:03.919 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:03.828+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:03.919 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:03.828+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:03.919 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:03.828+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:03.919 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:03.829+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:05.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:04.788+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:04.788+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:04.788+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:05.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:04.788+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:04.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:05.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:04.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:04.814+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:05.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:04.814+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:04.838+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:04.838+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:04.838+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:05.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:04.838+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:06.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:05.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:05.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:06.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:05.863+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:06.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:05.864+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:05.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:05.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:05.754+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:05.755+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:05.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:05.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:05.856+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:06.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:05.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:07.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:06.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:06.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:06.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:07.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:06.770+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:06.850+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:06.850+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:07.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:06.850+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:07.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:06.850+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:06.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:06.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:06.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:07.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:06.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:08.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:07.805+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:07.805+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:07.805+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:08.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:07.806+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:07.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:07.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:07.841+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:07.841+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:07.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:07.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:08.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:07.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:08.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:07.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:09.036 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:08.909+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32654 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:09.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:08.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:09.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:08.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:09.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:08.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:09.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:08.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:09.037 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:08.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:09.037 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:08.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:09.037 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:08.849+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:09.038 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:08.849+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:09.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:08.761+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:09.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:08.761+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:09.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:08.761+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:09.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:08.761+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:10.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:09.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:09.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:09.830+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:10.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:09.830+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:10.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:09.725+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:10.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:09.725+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:10.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:09.725+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:10.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:09.725+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:09.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:09.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:09.884+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:10.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:09.884+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:11.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:10.731+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:10.731+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:10.731+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:11.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:10.732+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:11.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:10.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:10.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:10.868+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:11.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:10.869+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:11.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:10.932+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:11.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:10.932+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:11.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:10.932+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:11.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:10.932+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:11.871 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:11.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:12.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:11.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:11.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:11.843+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:12.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:11.843+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:11.687+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:11.687+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:11.687+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:12.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:11.687+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:11.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:11.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:12.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:11.912+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:12.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:11.912+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:13.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:12.860+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:12.860+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:12.860+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:13.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:12.861+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:12.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:12.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:12.734+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:13.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:12.735+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:12.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:13.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:12.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:13.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:12.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:13.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:12.900+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:13.916 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:13.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:13.916 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:13.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:13.917 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:13.775+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:13.917 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:13.776+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:13.918 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:13.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:13.918 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:13.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:13.918 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:13.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:13.918 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:13.864+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:13.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:13.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:13.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:13.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:13.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:13.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:13.919 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:13.853+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:14.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:13.915+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32738 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:14.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:15.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:14.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:14.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:14.879+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:15.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:14.880+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:14.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:14.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:14.798+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:14.798+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:14.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:14.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:14.818+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:15.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:14.819+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:16.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:15.906+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:15.906+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:15.906+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:16.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:15.907+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:15.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:15.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:15.818+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:16.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:15.818+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:15.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:15.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:15.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:16.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:15.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:17.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:16.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:16.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:16.893+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:16.894+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:16.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:17.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:16.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:16.857+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:16.857+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:16.859+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:16.859+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:16.859+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:16.860+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:17.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:17.945 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:18.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:18.014 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:54:18.015 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-03-24T11:54:18.015 DEBUG:teuthology.orchestra.run.smithi100:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-03-24T11:54:18.034 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:18.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:18.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:17.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:18.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:17.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:18.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:17.852+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:18.146 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:17.852+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:18.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:17.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:18.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:17.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:18.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:17.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:18.147 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:17.852+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:18.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:17.909+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:18.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:17.909+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:18.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:17.909+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:18.148 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:17.909+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:19.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:18.920+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32820 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:19.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:18.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:18.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:18.840+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:18.840+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:18.821+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:18.821+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:19.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:18.821+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:19.144 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:18.821+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:18.944+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:18.944+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:18.944+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:19.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:18.944+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:20.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:19.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:20.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:19.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:19.782+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:20.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:19.782+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:19.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:19.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:19.803+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:20.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:19.804+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:19.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:19.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:19.946+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:20.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:19.946+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:21.023 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:20.923+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:21.024 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:20.923+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:21.024 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:20.923+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:21.024 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:20.923+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:21.024 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:20.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:21.024 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:20.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:21.024 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:20.768+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:21.025 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:20.768+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:21.025 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:20.762+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:21.025 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:20.762+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:21.025 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:20.762+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:21.025 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:20.762+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:21.090 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:22.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:21.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:21.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:21.732+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:22.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:21.733+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:21.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:21.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:21.769+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:22.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:21.769+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:21.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:21.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:21.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:21.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:23.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:22.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:22.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:22.741+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:23.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:22.742+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:22.779+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:22.779+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:22.779+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:23.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:22.779+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:22.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:22.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:22.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:23.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:22.872+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:23.925 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:23.704+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:23.925 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:23.704+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:23.926 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:23.704+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:23.926 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:23.704+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:23.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:23.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:23.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:23.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:23.926 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:23.811+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:23.927 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:23.811+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:23.927 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:23.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:23.927 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:23.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:23.927 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:23.878+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:23.927 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:23.878+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:24.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:23.925+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32903 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:25.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:24.894+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:25.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:24.894+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:24.894+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:25.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:24.894+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:24.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:24.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:24.721+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:25.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:24.722+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:24.850+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:24.850+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:24.850+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:25.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:24.850+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:26.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:25.711+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:26.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:25.711+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:25.711+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:26.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:25.712+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:25.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:25.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:26.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:25.847+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:26.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:25.848+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:25.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:25.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:25.940+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:26.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:25.940+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:27.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:26.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:27.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:26.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:26.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:27.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:26.697+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:26.892+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:26.892+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:26.892+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:27.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:26.892+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:26.896+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:26.896+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:26.896+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:27.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:26.897+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:28.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:27.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:28.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:27.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:28.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:27.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:28.044 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:27.855+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:28.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:27.912+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:28.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:27.912+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:28.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:27.912+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:28.045 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:27.912+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:28.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:27.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:28.045 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:27.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:28.046 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:27.712+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:28.046 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:27.712+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:28.977 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:28.930+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 32987 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:28.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:28.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:28.931+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:28.978 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:28.932+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:28.978 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:28.707+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:28.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:28.707+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:28.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:28.707+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:28.979 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:28.708+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:28.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:28.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:28.979 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:28.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:28.980 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:28.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:28.980 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:28.852+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:29.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:29.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:29.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:29.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:29.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:29.681+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:29.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:29.681+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:29.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:29.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:29.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:29.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:29.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:29.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:29.958 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:29.891+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:30.091 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:30.091 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:30.241 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:29.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:30.242 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:29.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:30.242 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:29.956+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:30.242 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:29.957+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:30.936 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:30.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:30.936 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:30.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:30.936 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:30.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:30.936 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:30.655+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:31.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:30.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:30.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:30.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:30.948+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:30.935+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:31.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:30.935+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:30.935+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:31.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:30.936+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:31.954 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:31.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:31.955 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:31.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:31.955 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:31.669+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:31.955 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:31.670+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:31.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:31.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:31.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:31.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:31.955 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:31.900+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:31.956 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:31.900+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:32.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:31.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:32.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:31.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:31.954+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:31.955+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:32.987 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:32.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:32.987 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:32.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:32.987 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:32.695+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:32.987 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:32.695+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:32.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:32.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:32.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:32.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:32.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:32.889+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:32.988 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:32.889+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:33.150 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:33.311 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:32.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:33.312 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:32.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:33.312 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:32.987+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:33.312 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:32.987+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:33.936 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:33.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:33.937 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:33.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:33.937 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:33.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:33.937 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:33.879+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:33.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:33.674+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:33.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:33.674+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:33.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:33.674+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:33.938 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:33.674+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:34.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:33.936+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33073 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:34.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:34.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:34.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:34.019+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:34.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:34.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:34.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:34.677+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:34.678+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:34.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:35.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:34.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:35.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:34.882+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:35.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:34.882+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:35.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:35.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:35.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:35.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:35.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:35.036+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:35.036+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:35.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:35.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:35.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:35.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:35.633+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:35.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:35.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:35.843+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:35.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:35.843+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:36.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:36.028+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:36.383 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:36.028+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:36.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:36.028+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:36.384 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:36.028+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:37.029 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:36.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:37.030 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:36.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:37.030 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:36.879+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:37.030 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:36.879+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:37.030 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:36.651+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:37.030 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:36.651+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:37.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:36.651+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:37.031 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:36.652+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:37.350 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:37.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:37.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:37.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:37.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:37.030+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:37.351 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:37.030+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:37.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:37.612+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:37.612+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:37.612+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:37.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:37.612+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:37.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:37.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:37.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:37.851+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:37.895 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:37.851+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:38.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:38.066+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:38.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:38.066+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:38.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:38.066+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:38.066+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:38.852 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:38.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:38.853 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:38.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:38.853 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:38.577+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:38.853 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:38.577+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:39.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:38.941+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33157 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:39.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:39.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:39.022+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:39.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:39.022+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:39.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:38.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:38.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:38.852+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:39.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:38.852+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:39.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:39.616+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:39.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:39.616+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:39.616+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:39.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:39.616+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:39.824+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:39.824+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:39.824+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:39.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:39.824+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:40.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:40.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:40.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:40.031+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:40.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:40.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:40.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:40.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:40.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:40.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:40.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:40.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:40.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:40.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:40.839+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:40.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:40.839+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:41.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:41.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:41.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:41.020+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:41.020+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:41.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:41.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:41.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:41.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:41.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:41.611+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:41.862 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:41.611+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:42.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:41.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:42.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:41.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:41.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:42.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:41.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:42.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:42.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:42.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:42.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:42.062+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:42.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:42.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:42.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:42.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:42.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:42.593+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:42.818 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:42.594+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:43.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:42.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:43.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:42.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:43.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:42.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:43.087 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:42.818+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:43.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:43.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:43.086+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:43.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:43.087+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:43.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:43.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:43.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:43.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:43.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:43.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:43.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:43.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:43.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:43.604+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:43.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:43.605+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:44.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:43.947+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33243 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:44.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:44.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:44.048+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:44.049+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:44.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:44.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:44.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:44.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:44.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:44.600+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:44.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:44.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:44.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:44.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:44.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:44.845+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:45.152 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:45.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:45.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:45.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:45.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:45.056+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:45.056+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:45.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:45.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:45.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:45.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:45.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:45.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:45.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:45.805+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:45.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:45.805+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:46.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:46.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:46.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:46.029+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:46.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:46.030+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:47.032 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:46.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:47.032 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:46.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:47.032 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:46.658+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:47.032 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:46.659+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:47.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:46.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:47.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:46.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:47.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:46.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:47.033 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:46.801+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:47.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.031+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:47.998 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:47.666+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:47.998 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:47.666+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:47.998 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:47.666+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:47.999 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:47.666+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:47.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:47.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:47.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:47.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:47.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:47.785+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:47.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:47.785+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:48.227 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:48.336 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:48.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:48.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:48.337 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:47.998+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:48.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:48.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:48.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:48.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:48.637+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:48.637+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:48.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:48.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:48.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:48.817+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:48.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:48.817+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:49.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:48.951+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33327 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:48.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:48.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:48.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:48.977+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:49.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:49.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:49.776+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:49.894 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:49.776+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:49.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:49.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:49.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:49.614+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:50.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:50.001+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:50.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:50.001+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:50.001+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:50.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:50.002+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:51.002 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:50.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:50.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:50.655+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:50.656+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:50.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:50.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:50.823+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:51.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:50.824+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:51.228 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:51.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:51.294 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:51.294 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:54:51.295 INFO:tasks.cephadm:Archiving crash dumps... 2024-03-24T11:54:51.297 DEBUG:teuthology.misc:Transferring archived files from smithi100:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869/remote/smithi100/crash 2024-03-24T11:54:51.297 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/crash -- . 2024-03-24T11:54:51.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:51.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:51.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:51.328 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.002+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:51.366 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/crash to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869/remote/smithi105/crash 2024-03-24T11:54:51.367 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:54:51.368 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:54:51.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:51.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:51.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:51.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:51.802+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:51.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:51.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:51.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:51.618+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:52.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:52.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:52.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:51.957+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:52.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:52.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:52.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:52.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:52.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:52.800+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:52.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:52.800+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:52.631+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:52.631+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:52.631+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:52.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:52.631+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:53.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:52.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:52.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:52.937+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:52.937+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:53.908 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:53.653+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:53.908 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:53.653+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:53.908 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:53.653+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:53.908 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:53.653+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:53.908 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:53.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:53.908 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:53.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:53.909 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:53.789+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:53.909 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:53.790+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:54.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:53.957+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33413 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:53.907+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:53.907+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:53.907+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:53.908+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:54.430 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:54:54.892 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:54.826+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:54.826+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:54.826+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:54.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:54.826+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:54.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:54.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:54.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:54.606+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:54.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:54.607+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:55.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:54.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:55.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:54.952+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:54.953+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:54.953+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:55.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:55.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:55.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:55.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:55.867 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:55.562+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:55.868 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:55.563+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:56.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:55.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:55.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:55.979+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:56.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:55.979+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:56.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:55.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:55.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:55.866+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:56.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:55.867+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:56.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:56.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:56.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:56.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:56.543+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:57.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:56.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:56.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:56.977+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:56.977+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:56.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:56.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:56.913+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:57.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:56.913+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:57.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:57.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:57.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:57.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:57.509+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:58.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:57.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:57.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:57.970+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:58.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:57.970+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:57.894+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:57.894+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:58.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:57.894+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:58.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:57.894+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:58.854 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:58.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:58.854 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:58.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:58.854 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:58.503+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:58.854 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:58.503+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:59.142 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:54:58.962+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33496 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:54:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:58.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:58.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:58.854+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:54:59.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:58.854+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:54:59.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:58.922+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:59.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:58.922+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:59.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:58.922+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:54:59.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:58.923+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:54:59.880 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:59.512+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:59.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:59.512+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:59.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:59.512+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:54:59.881 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:54:59.512+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:00.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:59.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:00.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:59.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:00.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:59.902+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:00.135 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:54:59.902+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:00.135 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:59.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:00.136 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:59.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:00.136 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:59.880+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:00.136 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:54:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:54:59.880+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:00.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:00.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:00.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:00.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:00.543+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:00.544+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:01.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:00.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:01.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:00.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:01.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:00.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:00.917+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:00.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:00.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:00.905+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:00.906+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:01.778 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:01.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:01.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:01.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:01.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:01.496+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:01.779 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:01.496+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:02.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:01.940+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:01.940+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:01.940+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:01.940+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:01.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:01.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:01.917+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:02.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:01.917+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:02.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:02.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:02.507+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:02.508+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:03.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:02.943+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:03.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:02.943+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:03.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:02.943+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:03.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:02.944+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:02.914+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:02.914+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:02.914+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:03.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:02.915+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:03.432 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:03.433 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:03.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:03.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:03.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:03.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:03.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:03.516+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:03.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:03.516+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:04.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:03.967+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33582 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:03.975+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:03.975+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:04.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:03.975+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:04.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:03.975+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:04.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:03.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:04.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:03.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:04.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:03.919+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:04.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:03.919+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:04.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:04.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:04.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:04.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:04.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:04.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:04.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:04.508+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:05.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:05.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:05.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:05.528 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.019+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:05.528 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:04.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:05.529 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:04.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:05.529 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:04.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:05.529 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:04.920+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:05.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:05.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:05.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:05.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:05.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:05.483+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:05.890 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:05.483+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:06.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:05.890+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:05.890+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:05.890+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:06.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:05.890+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:06.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.973+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:06.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:05.973+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:06.498 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:06.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:06.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:06.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:06.460+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:06.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:06.460+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:07.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:06.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:07.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:06.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:07.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:06.899+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:07.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:06.900+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:07.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:07.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:07.010+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:07.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:07.011+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:07.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:07.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:07.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:07.508+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:07.509+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:08.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:07.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:07.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:07.901+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:07.902+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:08.040+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:08.040+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:08.040+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:08.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:08.040+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:08.856 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:08.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:08.856 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:08.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:08.856 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:08.522+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:08.856 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:08.522+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:09.145 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:08.974+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33666 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:09.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:09.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:09.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:09.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:09.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:09.027+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:09.146 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:09.027+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:09.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:08.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:09.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:08.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:09.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:08.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:09.147 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:08.856+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:09.837 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:09.498+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:09.837 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:09.498+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:09.838 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:09.498+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:09.838 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:09.498+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:10.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:10.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:10.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:10.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:10.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:10.043+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:10.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:10.044+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:09.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:09.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:09.837+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:10.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:09.837+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:10.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:10.497+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:10.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:10.497+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:10.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:10.497+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:10.801 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:10.497+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:11.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:10.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:11.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:10.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:11.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:10.801+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:11.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:10.801+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:11.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:11.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:11.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:11.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:11.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:11.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:11.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:11.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:11.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:11.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:11.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:11.540+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:11.820 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:11.540+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:12.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:11.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:11.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:11.820+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:11.820+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:12.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:12.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:12.059+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:12.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:12.059+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:12.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:12.518+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:12.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:12.518+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:12.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:12.518+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:12.861 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:12.518+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:13.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:12.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:13.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:12.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:13.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:12.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:13.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:12.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:13.041+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:13.041+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:13.041+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:13.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:13.042+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:13.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:13.524+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:13.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:13.524+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:13.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:13.524+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:13.844 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:13.524+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:14.101 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:13.979+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33752 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:14.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:13.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:14.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:13.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:14.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:13.844+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:14.102 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:13.844+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:14.103 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:14.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:14.103 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:14.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:14.103 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:14.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:14.103 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:14.034+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:14.871 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:14.553+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:14.871 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:14.553+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:14.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:14.553+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:14.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:14.553+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:15.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:14.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:14.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:14.872+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:15.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:14.872+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:15.081+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:15.081+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:15.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:15.081+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:15.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:15.082+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:15.872 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:15.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:15.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:15.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:15.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:15.574+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:15.873 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:15.574+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:16.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:15.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:16.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:15.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:15.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:16.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:15.873+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:16.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:16.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:16.080+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:16.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:16.080+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:16.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:16.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:16.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:16.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:16.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:16.617+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:16.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:16.618+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:17.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:17.110+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:17.110+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:17.110+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:17.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:17.110+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:16.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:16.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:17.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:16.922+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:17.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:16.922+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:17.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:17.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:17.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:17.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:17.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:17.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:17.578+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:18.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:18.093+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:18.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:18.093+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:18.093+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:18.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:18.093+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:17.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:17.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:17.938+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:18.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:17.938+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:18.499 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:18.500 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:18.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:18.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:18.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:18.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:18.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:18.566+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:18.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:18.567+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:19.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:18.984+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33836 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:19.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:18.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:18.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:18.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:18.905+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:19.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:19.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:19.062+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:19.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:19.062+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:19.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:19.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:19.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:19.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:19.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:19.598+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:19.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:19.598+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:20.245 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:20.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:20.245 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:20.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:20.245 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:20.091+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:20.245 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:20.092+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:20.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:19.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:20.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:19.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:20.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:19.905+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:20.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:19.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:20.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:20.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:20.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:20.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:20.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:20.613+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:20.877 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:20.613+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:21.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:21.064+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:21.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:21.064+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:21.064+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:21.064+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:20.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:20.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:20.875+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:20.876+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:21.570 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:21.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:21.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:21.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:21.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:21.642+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:21.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:21.642+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:21.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:21.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:21.873+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:21.893 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:21.873+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:22.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:22.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:22.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:22.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:22.018+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:22.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:22.018+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:22.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:22.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:22.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:22.864 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:22.686+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:23.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:23.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:23.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:23.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:23.025+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:22.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:22.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:22.864+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:23.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:22.864+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:23.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:23.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:23.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:23.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:23.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:23.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:23.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:23.897+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:23.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:23.661+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:23.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:23.661+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:23.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:23.661+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:23.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:23.661+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:24.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:23.991+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 33922 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:24.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:24.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.008+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:24.572 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:24.573 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:24.643 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:24.644 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:55:24.644 INFO:tasks.cephadm:Checking cluster log for badness... 2024-03-24T11:55:24.645 DEBUG:teuthology.orchestra.run.smithi100:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-03-24T11:55:24.690 INFO:tasks.cephadm:Compressing logs... 2024-03-24T11:55:24.691 DEBUG:teuthology.orchestra.run.smithi100:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-03-24T11:55:24.734 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:24.735 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:24.766 INFO:teuthology.orchestra.run.smithi100.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-03-24T11:55:24.766 INFO:teuthology.orchestra.run.smithi100.stderr:: No such file or directory 2024-03-24T11:55:24.767 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.audit.log 2024-03-24T11:55:24.767 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.1.log 2024-03-24T11:55:24.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi100.ccifff.log 2024-03-24T11:55:24.767 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.audit.log: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.1.log: gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mon.smithi100.log 2024-03-24T11:55:24.768 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi100.begydc.log 2024-03-24T11:55:24.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi100.ccifff.log: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mon.smithi100.log: gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mgr.smithi100.smxgyi.log 2024-03-24T11:55:24.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi100.begydc.log: gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.0.log 2024-03-24T11:55:24.768 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mgr.smithi100.smxgyi.log: 90.0% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.audit.log.gz 2024-03-24T11:55:24.769 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.cephadm.log 2024-03-24T11:55:24.770 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.0.log: 93.3% -- replaced with /var/log/ceph/cephadm.log.gz 2024-03-24T11:55:24.771 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.log 2024-03-24T11:55:24.771 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.cephadm.log: gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.2.log 2024-03-24T11:55:24.772 INFO:teuthology.orchestra.run.smithi100.stderr: 81.7% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.cephadm.log.gz 2024-03-24T11:55:24.772 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.log: 86.1% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph.log.gz 2024-03-24T11:55:24.773 INFO:teuthology.orchestra.run.smithi100.stderr:gzip -5 --verbose -- /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-volume.log 2024-03-24T11:55:24.785 INFO:teuthology.orchestra.run.smithi100.stderr:/var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.2.log: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-volume.log: 90.9% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-volume.log.gz 2024-03-24T11:55:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:24.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:24.633+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:24.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:24.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:24.634+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:25.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.999+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:25.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:24.999+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:24.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:24.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:24.906+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:25.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:24.906+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:25.632 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mgr.smithi100.smxgyi.log: file size changed while zipping 2024-03-24T11:55:25.632 INFO:teuthology.orchestra.run.smithi100.stderr: 89.9% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mgr.smithi100.smxgyi.log.gz 2024-03-24T11:55:25.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:25.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:25.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:25.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:25.620+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:25.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:25.621+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:26.211 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mon.smithi100.log: file size changed while zipping 2024-03-24T11:55:26.211 INFO:teuthology.orchestra.run.smithi100.stderr: 92.8% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mon.smithi100.log.gz 2024-03-24T11:55:26.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:25.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:26.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:25.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:26.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:25.972+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:26.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:25.972+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:25.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:25.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:25.945+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:25.945+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:26.622 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.2.log: file size changed while zipping 2024-03-24T11:55:26.623 INFO:teuthology.orchestra.run.smithi100.stderr: 93.4% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.2.log.gz 2024-03-24T11:55:26.726 INFO:teuthology.orchestra.run.smithi100.stderr: 95.3% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi100.ccifff.log.gz 2024-03-24T11:55:26.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:26.666+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:26.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:26.666+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:26.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:26.666+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:26.970 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:26.666+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:27.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:26.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:27.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:26.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:27.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:26.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:27.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:26.970+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:26.973+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:26.973+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:26.973+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:27.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:26.973+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:27.535 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.1.log: file size changed while zipping 2024-03-24T11:55:27.535 INFO:teuthology.orchestra.run.smithi100.stderr: 92.6% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.1.log.gz 2024-03-24T11:55:27.751 INFO:teuthology.orchestra.run.smithi100.stderr:gzip: /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.0.log: file size changed while zipping 2024-03-24T11:55:27.751 INFO:teuthology.orchestra.run.smithi100.stderr: 91.9% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-osd.0.log.gz 2024-03-24T11:55:27.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:27.951 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:27.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:27.951 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:27.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:27.951 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:27.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:27.952 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:27.643+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:28.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:27.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:28.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:27.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:27.951+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:27.951+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:27.997+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:27.997+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:27.997+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:28.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:27.998+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:28.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:28.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:28.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:28.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:28.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:28.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:28.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:28.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:29.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:28.995+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 34006 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:29.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:28.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:28.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:28.921+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:29.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:28 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:28.921+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:29.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:29.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:29.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:29.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:29.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:29.007+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:29.933 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:29.644+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:29.933 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:29.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:29.933 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:29.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:29.933 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:29.645+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:30.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:29.932+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:30.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:29.932+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:30.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:29.932+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:30.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:29.933+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:30.246 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:30.247 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:30.247 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:30.247 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.006+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:30.937 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:30.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:30.938 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:30.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:30.938 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:30.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:30.938 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:30.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:31.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:30.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:31.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:30.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:31.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:30.936+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:31.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:30.936+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:31.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:30.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:31.949 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:31.674+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:31.949 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:31.674+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:31.949 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:31.674+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:31.949 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:31.674+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:32.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:31.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:31.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:31.963+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:32.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:31.963+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:32.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:31.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:31.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:31.948+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:32.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:31 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:31.949+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:32.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:32.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:32.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:32.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:32.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:32.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:32.993 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:32.675+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:32.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:32.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:32.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:32.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:32.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:32.920+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:32.993 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:32.920+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:33.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:32.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:33.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:32.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:33.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:32.992+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:33.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:32 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:32.992+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:34.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:33.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:34.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:33.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:34.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:33.907+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:34.003 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:33.907+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:34.003 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:33.710+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:34.004 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:33.710+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:34.004 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:33.710+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:34.004 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:33 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:33.710+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:34.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:34.000+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 34091 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:34.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:34.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:34.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:34.039+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:34.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:34.040+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:35.025 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:34.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:35.025 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:34.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:35.025 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:34.698+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:35.026 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:34.699+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:35.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:34.870+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:35.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:34.870+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:35.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:34.870+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:35.026 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:34 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:34.870+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:35.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:35.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:35.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:35.025+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:35.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:35.025+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:36.033 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:35.652+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:36.033 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:35.652+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:36.033 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:35.652+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:36.033 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:35.652+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:36.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:35.896+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:36.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:35.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:36.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:35.897+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:36.034 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:35 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:35.897+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:36.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:36.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:36.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:36.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:36.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:36.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:36.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:36.033+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:36.807 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:36.808 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:37.070 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:36.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:37.070 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:36.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:37.070 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:36.645+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:37.071 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:36.646+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:37.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:36.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:37.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:36.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:37.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:36.861+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:37.071 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:36 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:36.861+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:37.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:37.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:37.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:37.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:37.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:37.070+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:37.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:37.070+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:38.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:37.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:38.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:37.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:38.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:37.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:38.069 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:37.688+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:38.069 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:37.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:38.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:37.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:38.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:37.855+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:38.070 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:37 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:37.855+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:38.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:38.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:38.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:38.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:38.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:38.069+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:38.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:38.069+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:38.686+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:38.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:38.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:38.977 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:38.687+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:38.977 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:38.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:38.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:38.828+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:38.978 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:38 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:38.829+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:39.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:39.004+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 34166 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:39.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:39.100+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:39.100+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:39.100+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:39.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:39.100+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:39.874 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:40.054 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:39.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:40.054 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:39.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:40.054 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:39.819+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:40.054 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:39.819+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:40.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:39.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:40.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:39.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:40.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:39.715+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:40.055 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:39 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:39.716+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:40.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:40.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:40.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:40.054+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:40.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:40.054+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:41.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:40.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:41.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:40.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:41.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:40.675+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:41.008 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:40.676+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:41.008 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:40.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:41.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:40.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:41.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:40.832+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:41.009 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:40 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:40.832+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:41.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:41.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:41.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:41.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.008+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:41.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:41.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:41.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:41.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:41.991 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:41.676+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:41.992 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:41.677+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:41.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:41.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:41.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:41.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:41.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:41.871+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:41.992 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:41.871+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:42.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:42.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:42.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.991+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:42.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:41 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:41.991+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:43.034 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:42.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:43.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:42.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:43.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:42.643+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:43.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:42.644+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:43.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:42.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:43.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:42.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:43.036 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:42.860+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:43.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:42 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:42.860+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:43.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:43.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:43.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:43.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:43.034+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:43.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:43.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:43.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:43.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:43.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:43.603+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:43.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:43.603+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:44.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:44.008+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 34251 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:44.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:44.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:44.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:44.072+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:44.073+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:43.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:43.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:43.898+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:44.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:43 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:43.899+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:44.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:44.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:44.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:44.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:44.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:44.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:44.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:44.619+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:45.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:44.937+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:45.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:44.937+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:45.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:44.937+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:45.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:44 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:44.937+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:45.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:45.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:45.033+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:45.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:45.033+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:45.891 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:45.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:45.891 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:45.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:45.891 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:45.610+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:45.891 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:45.610+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:46.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:46.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:46.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:46.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:46.015+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:46.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:46.015+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:45.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:45.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:45.891+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:46.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:45 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:45.891+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:46.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:46.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:46.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:46.592+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:46.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:46.592+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:47.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:47.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:47.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:47.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.002+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:46.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:46.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:46.928+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:47.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:46 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:46.928+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:47.394 INFO:teuthology.orchestra.run.smithi100.stderr: 91.6% -- replaced with /var/log/ceph/e2f9ef6c-e9ce-11ee-95cd-87774f69a715/ceph-mds.cephfs.smithi100.begydc.log.gz 2024-03-24T11:55:47.395 INFO:teuthology.orchestra.run.smithi100.stderr: 2024-03-24T11:55:47.395 INFO:teuthology.orchestra.run.smithi100.stderr:real 0m22.648s 2024-03-24T11:55:47.395 INFO:teuthology.orchestra.run.smithi100.stderr:user 0m33.545s 2024-03-24T11:55:47.395 INFO:teuthology.orchestra.run.smithi100.stderr:sys 0m0.944s 2024-03-24T11:55:47.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:47.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:47.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:47.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:47.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:47.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:47.620+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:48.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:48.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:48.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:48.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:47.969+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:47.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:47.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:47.954+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:48.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:47 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:47.954+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:48.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:48.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:48.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:48.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:48.578+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:48.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:48.578+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:49.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:49.014+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 34335 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:48.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:48.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:48.971+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:49.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:48.971+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:48.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:48.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:48.987+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:49.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:48 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:48.987+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:49.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:49.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:49.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:49.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:49.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:49.896 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:49.618+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:50.247 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:49.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:50.247 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:49.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:50.248 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:49.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:50.248 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:49 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:49.993+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:50.248 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:50.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:50.248 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:50.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:50.248 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:50.006+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:50.249 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:50.007+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:50.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:50.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:50.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:50.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:50.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:50.641+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:50.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:50 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:50.641+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:51.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:51.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:51.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:51.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:51.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:51.006+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:51.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:51.007+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.031+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:51.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.031+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:51.875 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:51.876 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:51.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:51.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:51.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:51.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:51.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:51.618+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:51.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:51.618+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:52.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:52.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:52.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.990+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:52.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:51 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:51.991+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:52.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:52.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:52.044+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:52.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:52.044+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:53.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:52.648+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:53.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:52.648+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:53.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:52.648+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:53.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:52 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:52.648+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:53.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:53.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:53.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:53.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.036+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:53.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:53.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:53.057+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:53.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:53.058+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:53.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:53.621+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:53.621+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:53.621+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:53.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:53.622+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:54.392 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:54.020+0000 7fa3e9c55700 -1 mon.smithi100@0(probing) e2 get_health_metrics reporting 34421 slow ops, oldest is log(1 entries from seq 302 at 2024-03-24T11:19:01.183959+0000) 2024-03-24T11:55:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:54.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:54.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:54.065+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:54.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:54.065+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:54.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.988+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:54.394 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:53 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:53.988+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:54.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:54.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:54.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:54.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:54.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:54.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:54.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:54.623+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:54.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:55.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:55.059+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:55.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:55.059+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:55.059+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:55.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:55.059+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:54.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:54.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:54.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:55.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:54 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:54.965+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:55.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:55.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:55.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:55.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:55.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:55.589+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:55.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:55.590+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:56.255 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:55.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:56.255 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:55.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:56.256 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:55.927+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:56.256 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:55 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:55.927+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:56.256 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:56.045+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:56.256 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:56.045+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:56.256 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:56.045+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:56.256 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:56.045+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:56.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:56.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:56.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:56.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:56.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:56.599+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:56.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:56.599+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:57.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:56.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:57.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:56.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:56.947+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:57.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:56 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:56.947+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:57.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:57.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:57.061+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:57.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:57.061+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:57.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:57.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:57.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:57.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:57.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:57.619+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:57.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:57.619+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:57.943 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:55:57.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:55:58.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:55:58.014 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:55:58.015 INFO:tasks.cephadm:Removing cluster... 2024-03-24T11:55:58.015 DEBUG:teuthology.orchestra.run.smithi100:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 --force 2024-03-24T11:55:58.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:57.930+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:58.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:57.930+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:58.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:57.930+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:58.305 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:57 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:57.930+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:58.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:58.055+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:58.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:58.055+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:58.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:58.055+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:58.306 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:58.056+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:58.633 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:58 smithi100 systemd[1]: Stopping Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:55:58.634 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:58.544+0000 7fa3ecc5b700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi100 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:55:58.634 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100[28937]: 2024-03-24T11:55:58.544+0000 7fa3ecc5b700 -1 mon.smithi100@0(probing) e2 *** Got Signal Terminated *** 2024-03-24T11:55:58.892 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:58 smithi100 podman[86102]: 2024-03-24 11:55:58.882054584 +0000 UTC m=+0.436046531 container died f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240319, RELEASE=quincy-9c4e0c3, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, ceph=True) 2024-03-24T11:55:58.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:58.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:58.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:58.632+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:58.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:58.633+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:59.245 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:58.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:59.246 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:58.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:59.246 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:58.978+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:55:59.246 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:55:58 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:55:58.978+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:55:59.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:59.046+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:59.246 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:59.046+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:59.247 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:59.046+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:55:59.247 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:55:59.046+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:55:59.522 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:59 smithi100 podman[86102]: 2024-03-24 11:55:59.246742255 +0000 UTC m=+0.800734202 container cleanup f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-03-24T11:55:59.522 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:59 smithi100 bash[86102]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100 2024-03-24T11:55:59.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:59 smithi100 podman[86116]: 2024-03-24 11:55:59.523075934 +0000 UTC m=+0.634351134 container remove f7fbf127ce8adb5701a4b945cee247d216902875d2ecd2ece968a4bf3fa52245 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-mon-smithi100, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-9c4e0c3, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, ceph=True) 2024-03-24T11:55:59.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:59 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service: Deactivated successfully. 2024-03-24T11:55:59.893 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:59 smithi100 systemd[1]: Stopped Ceph mon.smithi100 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:55:59.894 INFO:journalctl@ceph.mon.smithi100.smithi100.stdout:Mar 24 11:55:59 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@mon.smithi100.service: Consumed 26.686s CPU time. 2024-03-24T11:55:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:59.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:59.894 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:59.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:59.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:59.625+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:55:59.895 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:55:59 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:55:59.625+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:00.248 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:00.248 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:00.248 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.002+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:00.249 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.002+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:00.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:00.097+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:00.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:00.097+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:00.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:00.097+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:00.249 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:00.097+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:00.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:00.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:00.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:00.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:00.634+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:00.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:00.635+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:01.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.964+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:01.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:00 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:00.964+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:01.102+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:01.102+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:01.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:01.102+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:01.394 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:01.102+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:01.999 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:01.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:01.999 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:01.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:01.999 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:01.673+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:01.999 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:01 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:01.673+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:02.295 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:01.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:02.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:01.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:02.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:01.998+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:02.296 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:01.998+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:02.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:02.056+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:02.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:02.056+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:02.296 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:02.056+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:02.297 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:02.056+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:02.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:02.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:02.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:02.635+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:02.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:02 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:02.635+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:03.223 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:03.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:03.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:03.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:03.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:03.034+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:03.224 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:03.035+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:03.224 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:03.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:03.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:03.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:03.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:03.083+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:03.225 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:03.083+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:03.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:03.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:03.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:03.623+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:03.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:03.624+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:03.883 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:03 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:03.624+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:04.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:04.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:04.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:04.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:04.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:04.073+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:04.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:04.074+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:04.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:04.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:04.049+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:04.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:04.049+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:04.966 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:04.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:04.966 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:04.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:04.966 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:04.659+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:04.966 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:04 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:04.660+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:05.219 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:05.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:05.219 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:05.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:05.220 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:05.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:05.220 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:05.063+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:05.220 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:05.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:05.220 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:05.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:05.220 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:05.067+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:05.220 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:05.068+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:05.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:05.671+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:05.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:05.671+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:05.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:05.671+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:05.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:05 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:05.671+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:06.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:06.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:06.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:06.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:06.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:06.035+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:06.317 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:06.036+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:06.317 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:06.102+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:06.318 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:06.102+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:06.318 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:06.102+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:06.318 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:06.102+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:06.973 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:06.660+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:06.973 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:06.660+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:06.973 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:06.660+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:06.974 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:06 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:06.661+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:07.263 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:07.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:07.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:07.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:07.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:07.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:07.264 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:07.065+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:07.264 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:07.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:07.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.007+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:07.265 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.007+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:07.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:07.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:07.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:07.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:07.892 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:07.701+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:07.893 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:07.702+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:08.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:08.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:08.392 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.969+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:08.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:07 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:07.970+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:08.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:08.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:08.076+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:08.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:08.076+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:09.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:08.965+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:09.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:08.965+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:09.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:08.965+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:09.035 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:08.965+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:09.035 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:08.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:09.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:08.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:09.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:08.744+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:09.036 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:08 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:08.745+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:09.305 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 systemd[1]: Stopping Ceph osd.0 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:56:09.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:09.078+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:09.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:09.078+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:09.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:09.079+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:09.305 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:09.079+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:09.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.305+0000 7f1f77faa700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:56:09.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.305+0000 7f1f77faa700 -1 osd.0 46 *** Got signal Terminated *** 2024-03-24T11:56:09.642 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.305+0000 7f1f77faa700 -1 osd.0 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:56:10.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:10.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:10.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.759+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:10.105 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:09.759+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:10.106 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:09.967+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:10.106 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:09.967+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:10.106 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:09.967+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:10.106 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:09 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:09.968+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:10.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:10.105+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:10.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:10.105+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:10.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:10.105+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:10.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:10.105+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:10.984 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:10.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:10.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:10.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:10.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:10.724+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:10.985 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:10.724+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:11.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:11.126+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:11.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:11.126+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:11.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:11.126+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:11.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:11.126+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:10.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:10.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:10.984+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:11.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:10 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:10.985+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:11.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:11.690+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:11.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:11.690+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:11.957 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:11.690+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:11.958 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:11.690+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:12.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:12.094+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:12.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:12.094+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:12.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:12.094+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:12.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:12.094+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:11.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:11.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:11.957+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:12.393 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:11 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:11.957+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:13.118 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:12.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:13.118 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:12.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:13.118 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:12.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:13.118 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:12.916+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:13.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:12.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:13.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:12.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:13.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:12.696+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:13.119 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:12 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:12.696+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:13.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:13.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:13.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:13.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:13.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:13.117+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:13.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:13.118+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:14.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:13.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:14.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:13.687+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:14.142 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:13.688+0000 7f1f735a6700 -1 osd.0 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:40.747145+0000 (oldest deadline 2024-03-24T11:51:00.747145+0000) 2024-03-24T11:56:14.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0[47707]: 2024-03-24T11:56:13.688+0000 7f1f735a6700 -1 osd.0 46 get_health_metrics reporting 127 slow ops, oldest is osd_op(client.24297.0:10044 3.17 3:e8e03df1:::10000005893.00000000:head [write 0~8353 [1@-1] in=8353b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:13.911+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:13.911+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:13.911+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:14.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:13 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:13.912+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:14.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:14.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:14.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:14.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:14.642 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:14.144+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:14.643 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:14.145+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:15.088 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:14 smithi100 podman[86950]: 2024-03-24 11:56:14.775550723 +0000 UTC m=+5.602297048 container died d8dbc515015ae2c7cffdbe4f10cfad0383c7ab2031e81db1629c85220a727660 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, io.buildah.version=1.33.5, org.label-schema.build-date=20240319, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-9c4e0c3) 2024-03-24T11:56:15.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:14.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:15.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:14.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:15.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:14.916+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:15.089 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:14 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:14.916+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:15.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:15 smithi100 podman[86950]: 2024-03-24 11:56:15.180716625 +0000 UTC m=+6.007462950 container cleanup d8dbc515015ae2c7cffdbe4f10cfad0383c7ab2031e81db1629c85220a727660 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD) 2024-03-24T11:56:15.392 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:15 smithi100 bash[86950]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0 2024-03-24T11:56:15.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:15.108+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:15.108+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:15.108+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:15.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:15.108+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:15.805 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:15 smithi100 podman[86972]: 2024-03-24 11:56:15.398470255 +0000 UTC m=+0.625516402 container remove d8dbc515015ae2c7cffdbe4f10cfad0383c7ab2031e81db1629c85220a727660 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0, org.label-schema.schema-version=1.0, ceph=True, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , RELEASE=quincy-9c4e0c3, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T11:56:16.081 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:15.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:16.082 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:15.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:16.082 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:15.867+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:16.082 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:15 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:15.868+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:16.082 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:15 smithi100 podman[87063]: 2024-03-24 11:56:15.708750812 +0000 UTC m=+0.017540789 image pull 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232 2024-03-24T11:56:16.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:15 smithi100 podman[87063]: 2024-03-24 11:56:15.816057161 +0000 UTC m=+0.124847166 container create 0c680b155852637d06e9cfb483197885a2d5fcf0a73a7573718a2a4e377a0ae0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_BRANCH=HEAD, ceph=True, GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319) 2024-03-24T11:56:16.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 podman[87063]: 2024-03-24 11:56:16.027236507 +0000 UTC m=+0.336026477 container init 0c680b155852637d06e9cfb483197885a2d5fcf0a73a7573718a2a4e377a0ae0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3, ceph=True, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_BRANCH=HEAD) 2024-03-24T11:56:16.083 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 podman[87063]: 2024-03-24 11:56:16.030422979 +0000 UTC m=+0.339212942 container start 0c680b155852637d06e9cfb483197885a2d5fcf0a73a7573718a2a4e377a0ae0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, ceph=True, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux ) 2024-03-24T11:56:16.354 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 podman[87063]: 2024-03-24 11:56:16.120432254 +0000 UTC m=+0.429222240 container attach 0c680b155852637d06e9cfb483197885a2d5fcf0a73a7573718a2a4e377a0ae0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0-deactivate, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, org.label-schema.name=CentOS Stream 8 Base Image, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-03-24T11:56:16.354 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 podman[87063]: 2024-03-24 11:56:16.321509192 +0000 UTC m=+0.630299155 container died 0c680b155852637d06e9cfb483197885a2d5fcf0a73a7573718a2a4e377a0ae0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-03-24T11:56:16.354 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:16.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:16.354 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:16.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:16.355 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:16.081+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:16.355 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:16.081+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 podman[87091]: 2024-03-24 11:56:16.888943242 +0000 UTC m=+0.561820467 container remove 0c680b155852637d06e9cfb483197885a2d5fcf0a73a7573718a2a4e377a0ae0 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-0-deactivate, maintainer=Guillaume Abrioux , GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True) 2024-03-24T11:56:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.0.service: Deactivated successfully. 2024-03-24T11:56:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 systemd[1]: Stopped Ceph osd.0 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:56:17.143 INFO:journalctl@ceph.osd.0.smithi100.stdout:Mar 24 11:56:16 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.0.service: Consumed 25.686s CPU time. 2024-03-24T11:56:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:16.907+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:16.907+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:16.907+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:16 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:16.907+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:17.144 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 systemd[1]: Stopping Ceph osd.1 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:56:17.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:17.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:17.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:17.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:17.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:17.071+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:17.145 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:17.071+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:17.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.375+0000 7fc6dc43d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:56:17.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.375+0000 7fc6dc43d700 -1 osd.1 46 *** Got signal Terminated *** 2024-03-24T11:56:17.642 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.375+0000 7fc6dc43d700 -1 osd.1 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:56:18.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.870+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:18.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.870+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:18.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.870+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:18.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:17 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:17.870+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:18.063+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:18.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:18.064+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:18.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:18.064+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:19.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:18.854+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:19.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:18.854+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:18.854+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:19.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:18 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:18.854+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:19.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:19.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:19.021+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:19.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:19.021+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:20.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:19.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:20.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:19.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:20.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:19.852+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:20.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:19 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:19.853+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:20.001+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:20.001+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:20.001+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:20.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:20.001+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:21.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:20.836+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:21.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:20.836+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:20.836+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:21.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:20 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:20.837+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:21.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:21.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:21.051+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:21.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:21.051+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:22.142 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:21.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:21.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:21.796+0000 7fc6d6236700 -1 osd.1 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:44.266978+0000 (oldest deadline 2024-03-24T11:51:04.266978+0000) 2024-03-24T11:56:22.143 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:21 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1[55384]: 2024-03-24T11:56:21.796+0000 7fc6d6236700 -1 osd.1 46 get_health_metrics reporting 92 slow ops, oldest is osd_op(mds.0.4:730 2.6 2:654134d2:::mds0_openfiles.0:head [omap-set-header in=42b,omap-set-vals in=142403b,omap-rm-keys in=29314b] snapc 0=[] ondisk+write+known_if_redirected+full_force+supports_pool_eio e46) 2024-03-24T11:56:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:22.024+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:22.024+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:22.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:22.024+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:22.144 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:22 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:22.024+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:23.019 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:22 smithi100 podman[87180]: 2024-03-24 11:56:22.737405753 +0000 UTC m=+5.480841869 container died 8c2b1a04b4bfa5425b3203331badbc1b2ea0a6a53971bcbee65d8343a156f4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1, org.label-schema.name=CentOS Stream 8 Base Image, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-9c4e0c3, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.vendor=CentOS) 2024-03-24T11:56:23.276 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87180]: 2024-03-24 11:56:23.025642377 +0000 UTC m=+5.769078496 container cleanup 8c2b1a04b4bfa5425b3203331badbc1b2ea0a6a53971bcbee65d8343a156f4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-9c4e0c3, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:56:23.276 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 bash[87180]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1 2024-03-24T11:56:23.276 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:23.020+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:23.276 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:23.020+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:23.276 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:23.020+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:23.277 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:23 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:23.020+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:23.585 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87202]: 2024-03-24 11:56:23.276842002 +0000 UTC m=+0.535055871 container remove 8c2b1a04b4bfa5425b3203331badbc1b2ea0a6a53971bcbee65d8343a156f4b4 (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1, org.label-schema.build-date=20240319, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-9c4e0c3, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, ceph=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS) 2024-03-24T11:56:23.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87297]: 2024-03-24 11:56:23.586083485 +0000 UTC m=+0.110296371 container create be13f7678b967728d62c0917faedfba4616e911cf941b8465de86f2e7fd0d03e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, RELEASE=quincy-9c4e0c3, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, org.label-schema.name=CentOS Stream 8 Base Image) 2024-03-24T11:56:23.892 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87297]: 2024-03-24 11:56:23.493566377 +0000 UTC m=+0.017779241 image pull 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232 2024-03-24T11:56:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87297]: 2024-03-24 11:56:23.813525719 +0000 UTC m=+0.337738572 container init be13f7678b967728d62c0917faedfba4616e911cf941b8465de86f2e7fd0d03e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-9c4e0c3, org.label-schema.build-date=20240319, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, GIT_CLEAN=True, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2) 2024-03-24T11:56:23.893 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87297]: 2024-03-24 11:56:23.816623912 +0000 UTC m=+0.340836759 container start be13f7678b967728d62c0917faedfba4616e911cf941b8465de86f2e7fd0d03e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=) 2024-03-24T11:56:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:24.053+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:24.053+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:24.053+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:24.395 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:24 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:24.053+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:24.395 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:23 smithi100 podman[87297]: 2024-03-24 11:56:23.896579795 +0000 UTC m=+0.420792641 container attach be13f7678b967728d62c0917faedfba4616e911cf941b8465de86f2e7fd0d03e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1-deactivate, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , RELEASE=quincy-9c4e0c3, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS) 2024-03-24T11:56:24.396 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:24 smithi100 podman[87297]: 2024-03-24 11:56:24.09673839 +0000 UTC m=+0.620951234 container died be13f7678b967728d62c0917faedfba4616e911cf941b8465de86f2e7fd0d03e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1-deactivate, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD) 2024-03-24T11:56:25.037 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:24 smithi100 systemd[1]: Stopping Ceph osd.2 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715... 2024-03-24T11:56:25.037 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:24 smithi100 podman[87323]: 2024-03-24 11:56:24.742218435 +0000 UTC m=+0.640433929 container remove be13f7678b967728d62c0917faedfba4616e911cf941b8465de86f2e7fd0d03e (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, GIT_BRANCH=HEAD, ceph=True, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, RELEASE=quincy-9c4e0c3, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux ) 2024-03-24T11:56:25.037 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:24 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.1.service: Deactivated successfully. 2024-03-24T11:56:25.037 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:24 smithi100 systemd[1]: Stopped Ceph osd.1 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:56:25.038 INFO:journalctl@ceph.osd.1.smithi100.stdout:Mar 24 11:56:24 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.1.service: Consumed 26.169s CPU time. 2024-03-24T11:56:25.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:25.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.036+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.037+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.220+0000 7fd4f7a0d700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-03-24T11:56:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.220+0000 7fd4f7a0d700 -1 osd.2 46 *** Got signal Terminated *** 2024-03-24T11:56:25.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:25 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:25.220+0000 7fd4f7a0d700 -1 osd.2 46 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-03-24T11:56:26.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:26.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:26.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.038+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:26.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:26 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.038+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:27.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.999+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:27.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.999+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:27.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.999+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:27.143 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:26.999+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:28.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:27.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:28.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:27.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:28.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:27.993+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:28.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:27 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:27.993+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:29.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:29.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:29.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:29.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:29.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:29.010+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:29.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:29 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:29.010+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:30.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:30.008+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6804 osd.3 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:30.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:30.008+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6812 osd.4 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:30.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:30.008+0000 7fd4f3009700 -1 osd.2 46 heartbeat_check: no reply from 172.21.15.105:6820 osd.5 ever on either front or back, first ping sent 2024-03-24T11:50:43.097995+0000 (oldest deadline 2024-03-24T11:51:03.097995+0000) 2024-03-24T11:56:30.393 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:30 smithi100 ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2[63192]: 2024-03-24T11:56:30.008+0000 7fd4f3009700 -1 osd.2 46 get_health_metrics reporting 47 slow ops, oldest is osd_op(client.24297.0:10043 3.7 3:e1a372cd:::10000005892.00000000:head [write 0~1015 [1@-1] in=1015b] snapc 1=[] ondisk+write+known_if_redirected e46) 2024-03-24T11:56:31.052 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:30 smithi100 podman[87401]: 2024-03-24 11:56:30.738457181 +0000 UTC m=+5.633313222 container died e04c18a8a0097df9b5a60492d78b1c2bf331206788d8b0d9a8d78348f21578af (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_BRANCH=HEAD, ceph=True, RELEASE=quincy-9c4e0c3, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5) 2024-03-24T11:56:31.053 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87401]: 2024-03-24 11:56:31.053566196 +0000 UTC m=+5.948422243 container cleanup e04c18a8a0097df9b5a60492d78b1c2bf331206788d8b0d9a8d78348f21578af (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, RELEASE=quincy-9c4e0c3, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=) 2024-03-24T11:56:31.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 bash[87401]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2 2024-03-24T11:56:31.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87431]: 2024-03-24 11:56:31.271551819 +0000 UTC m=+0.530527604 container remove e04c18a8a0097df9b5a60492d78b1c2bf331206788d8b0d9a8d78348f21578af (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2, org.label-schema.vendor=CentOS, ceph=True, RELEASE=quincy-9c4e0c3, org.label-schema.build-date=20240319, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, io.buildah.version=1.33.5, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-03-24T11:56:31.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87524]: 2024-03-24 11:56:31.578412966 +0000 UTC m=+0.017608078 image pull 837124da804ffab28571a3e286aff1c534a701f228c0d3d7a309fcb128aca1de quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232 2024-03-24T11:56:31.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87524]: 2024-03-24 11:56:31.688940193 +0000 UTC m=+0.128135289 container create 38b11b467e0a45fae09759543d20f7e846298e73a46d808b22bca9f332fb513c (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2-deactivate, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240319, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, RELEASE=quincy-9c4e0c3, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , ceph=True, GIT_CLEAN=True) 2024-03-24T11:56:31.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87524]: 2024-03-24 11:56:31.916832081 +0000 UTC m=+0.356027187 container init 38b11b467e0a45fae09759543d20f7e846298e73a46d808b22bca9f332fb513c (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.build-date=20240319, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:56:31.999 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87524]: 2024-03-24 11:56:31.919858983 +0000 UTC m=+0.359054080 container start 38b11b467e0a45fae09759543d20f7e846298e73a46d808b22bca9f332fb513c (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2-deactivate, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3, GIT_CLEAN=True, io.buildah.version=1.33.5, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.build-date=20240319, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30) 2024-03-24T11:56:32.000 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:31 smithi100 podman[87524]: 2024-03-24 11:56:31.999782428 +0000 UTC m=+0.438977524 container attach 38b11b467e0a45fae09759543d20f7e846298e73a46d808b22bca9f332fb513c (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2-deactivate, ceph=True, org.label-schema.schema-version=1.0, io.buildah.version=1.33.5, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, org.label-schema.name=CentOS Stream 8 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, RELEASE=quincy-9c4e0c3) 2024-03-24T11:56:32.392 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:32 smithi100 podman[87524]: 2024-03-24 11:56:32.19439185 +0000 UTC m=+0.633586946 container died 38b11b467e0a45fae09759543d20f7e846298e73a46d808b22bca9f332fb513c (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-9c4e0c3, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240319, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 8 Base Image, org.label-schema.license=GPLv2, ceph=True, org.label-schema.vendor=CentOS) 2024-03-24T11:56:32.736 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:32 smithi100 podman[87550]: 2024-03-24 11:56:32.711452422 +0000 UTC m=+0.512086504 container remove 38b11b467e0a45fae09759543d20f7e846298e73a46d808b22bca9f332fb513c (image=quay.ceph.io/ceph-ci/ceph@sha256:e2ff0b9ff70b30131a16fd0ac4779c04eb8f80752618b4f4b744a02826313232, name=ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715-osd-2-deactivate, io.buildah.version=1.33.5, maintainer=Guillaume Abrioux , RELEASE=quincy-9c4e0c3, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 8 Base Image, GIT_BRANCH=HEAD, org.label-schema.build-date=20240319, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=08b1ab54e11946f162e232e2750e80887689df30, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0) 2024-03-24T11:56:33.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:32 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.2.service: Deactivated successfully. 2024-03-24T11:56:33.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:32 smithi100 systemd[1]: Stopped Ceph osd.2 for e2f9ef6c-e9ce-11ee-95cd-87774f69a715. 2024-03-24T11:56:33.142 INFO:journalctl@ceph.osd.2.smithi100.stdout:Mar 24 11:56:32 smithi100 systemd[1]: ceph-e2f9ef6c-e9ce-11ee-95cd-87774f69a715@osd.2.service: Consumed 23.716s CPU time. 2024-03-24T11:56:35.796 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:56:35.798 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:56:38.878 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:56:47.880 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:56:47.881 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:56:50.942 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:02.944 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:02.944 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:06.014 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:09.015 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:09.016 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:09.086 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:09.086 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:57:09.087 INFO:tasks.cephadm:Teardown complete 2024-03-24T11:57:09.087 ERROR:teuthology.run_tasks:Manager failed: cephadm Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1553, in distribute_config_and_admin_keyring yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1807, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1810, in task log.info('Teardown begin') File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 1686, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 142, in download_cephadm _rm_cluster(ctx, cluster_name) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephadm.py", line 310, in _rm_cluster ctx.cluster.run(args=[ File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T11:57:09.109 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-03-24T11:57:09.123 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host 2024-03-24T11:57:09.143 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-03-24T11:57:09.143 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-03-24T11:57:09.157 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:09.157 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:12.226 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:21.227 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:21.228 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:24.290 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:36.291 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:36.292 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:39.358 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:42.359 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:42.359 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:42.434 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:42.434 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:57:42.491 ERROR:teuthology.run_tasks:Manager failed: install Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 148, in ship_utilities yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 626, in task yield File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/__init__.py", line 220, in install yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 150, in ship_utilities _remove_utilities(ctx, filenames) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/install/util.py", line 121, in _remove_utilities ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T11:57:42.512 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-03-24T11:57:42.522 INFO:teuthology.task.clock:Checking final clock skew... 2024-03-24T11:57:42.523 DEBUG:teuthology.orchestra.run.smithi100:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-03-24T11:57:42.531 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:42.532 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:42.546 INFO:teuthology.orchestra.run.smithi100.stderr:bash: line 1: ntpq: command not found 2024-03-24T11:57:42.550 INFO:teuthology.orchestra.run.smithi100.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-03-24T11:57:42.550 INFO:teuthology.orchestra.run.smithi100.stdout:=============================================================================== 2024-03-24T11:57:42.550 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 +889us[ +889us] +/- 66ms 2024-03-24T11:57:42.550 INFO:teuthology.orchestra.run.smithi100.stdout:^* hv02.front.sepia.ceph.com 3 7 377 41 +102us[ +102us] +/- 67ms 2024-03-24T11:57:42.551 INFO:teuthology.orchestra.run.smithi100.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 39 -824us[ -824us] +/- 70ms 2024-03-24T11:57:42.551 INFO:teuthology.orchestra.run.smithi100.stdout:^? hv04.front.sepia.ceph.com 0 9 0 - +0ns[ +0ns] +/- 0ns 2024-03-24T11:57:45.598 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:57:54.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:57:54.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:57:57.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:09.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:09.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:12.734 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:15.735 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:15.736 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:15.806 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:15.807 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:58:15.807 ERROR:teuthology.run_tasks:Manager failed: clock Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 66, in task yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/clock.py", line 72, in task cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T11:58:15.824 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-03-24T11:58:15.835 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-03-24T11:58:15.836 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-03-24T11:58:15.846 DEBUG:teuthology.orchestra.run.smithi100:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-03-24T11:58:15.934 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:15.935 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:19.006 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:28.007 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:28.008 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:31.070 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:43.071 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:43.072 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:46.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:49.143 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:49.144 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:49.214 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:58:49.214 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:58:49.215 ERROR:teuthology.run_tasks:Manager failed: selinux Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/__init__.py", line 136, in __exit__ self.teardown() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 165, in teardown self.archive_log() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/selinux.py", line 195, in archive_log self.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T11:58:49.223 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-03-24T11:58:49.236 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-03-24T11:58:49.246 INFO:teuthology.task.internal:Duration was 3409.401542 seconds 2024-03-24T11:58:49.247 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-03-24T11:58:49.257 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-03-24T11:58:49.257 DEBUG:teuthology.orchestra.run.smithi100:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-03-24T11:58:49.263 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:58:49.264 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:58:49.313 INFO:teuthology.orchestra.run.smithi100.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-03-24T11:58:52.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:01.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:01.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:04.386 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:16.387 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:16.388 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:19.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:22.460 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:22.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:22.530 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:22.530 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:59:22.531 ERROR:teuthology.run_tasks:Manager failed: internal.syslog Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 76, in syslog yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/syslog.py", line 85, in syslog cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T11:59:22.547 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-03-24T11:59:22.561 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-03-24T11:59:22.561 DEBUG:teuthology.orchestra.run.smithi100:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-03-24T11:59:22.599 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:22.600 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:25.662 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:34.663 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:34.664 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:37.726 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:49.727 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:49.728 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:52.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:55.800 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:55.801 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:55.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T11:59:55.871 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T11:59:55.871 ERROR:teuthology.run_tasks:Manager failed: internal.sudo Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 452, in sudo yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 455, in sudo ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T11:59:55.887 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-03-24T11:59:55.899 DEBUG:teuthology.orchestra.run.smithi100:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-03-24T11:59:55.906 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T11:59:55.907 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T11:59:55.934 INFO:teuthology.orchestra.run.smithi100.stdout:kernel.core_pattern = core 2024-03-24T11:59:58.974 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:00:07.975 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:00:07.977 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:00:11.038 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:00:23.039 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:00:23.040 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:00:26.110 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:00:29.111 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:00:29.112 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:00:29.182 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:00:29.182 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T12:00:29.183 ERROR:teuthology.run_tasks:Manager failed: internal.coredump Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 488, in coredump yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 492, in coredump cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T12:00:29.199 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-03-24T12:00:29.215 INFO:teuthology.task.internal:Transferring archived files... 2024-03-24T12:00:29.216 DEBUG:teuthology.misc:Transferring archived files from smithi100:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869/remote/smithi100 2024-03-24T12:00:29.216 DEBUG:teuthology.orchestra.run.smithi100:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-03-24T12:00:29.259 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869/remote/smithi105 2024-03-24T12:00:29.260 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:00:29.260 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:00:32.318 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:00:41.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:00:41.321 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:00:44.382 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:00:56.383 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:00:56.385 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:00:59.458 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:01:02.459 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:01:02.460 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:01:02.526 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:01:02.527 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T12:01:02.527 ERROR:teuthology.run_tasks:Manager failed: internal.archive Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 392, in archive yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 418, in archive misc.pull_directory(rem, archive_dir, path, maybe_compress) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/misc.py", line 748, in pull_directory r = remote.get_tar_stream(remotedir, sudo=True, compress=False) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 678, in get_tar_stream return self.run(args=args, wait=False, stdout=run.PIPE) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T12:01:02.553 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-03-24T12:01:02.567 INFO:teuthology.task.internal:Not uploading archives. 2024-03-24T12:01:02.567 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-03-24T12:01:02.579 INFO:teuthology.task.internal:Tidying up after the test... 2024-03-24T12:01:02.579 DEBUG:teuthology.orchestra.run.smithi100:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-03-24T12:01:02.589 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:01:02.590 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:01:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: 265594 4 drwxr-xr-x 3 ubuntu ubuntu 4096 Mar 24 11:57 /home/ubuntu/cephtest 2024-03-24T12:01:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: 265595 4 drwxr-xr-x 4 ubuntu ubuntu 4096 Mar 24 11:59 /home/ubuntu/cephtest/archive 2024-03-24T12:01:02.603 INFO:teuthology.orchestra.run.smithi100.stdout: 265613 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:01 /home/ubuntu/cephtest/archive/syslog 2024-03-24T12:01:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: 265663 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 24 11:01 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-03-24T12:01:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: 265626 0 -rw-rw-rw- 1 ubuntu ubuntu 0 Mar 24 11:01 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-03-24T12:01:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: 413340 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Mar 24 11:58 /home/ubuntu/cephtest/archive/audit 2024-03-24T12:01:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: 413342 176 -rw------- 1 ubuntu ubuntu 179815 Mar 24 11:58 /home/ubuntu/cephtest/archive/audit/audit.log.gz 2024-03-24T12:01:02.604 INFO:teuthology.orchestra.run.smithi100.stdout: 276575 368 -rwxr-xr-x 1 ubuntu ubuntu 375153 Mar 24 11:08 /home/ubuntu/cephtest/cephadm 2024-03-24T12:01:02.605 INFO:teuthology.orchestra.run.smithi100.stderr:rmdir: failed to remove '/home/ubuntu/cephtest': Directory not empty 2024-03-24T12:01:05.666 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:01:14.667 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:01:14.668 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:01:17.730 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:01:29.731 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:01:29.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:01:32.798 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:01:35.799 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-03-24T12:01:35.800 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-03-24T12:01:35.870 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-03-24T12:01:35.870 WARNING:teuthology.contextutil:'reconnect to {self.shortname}' reached maximum tries (5) after waiting for 30 seconds 2024-03-24T12:01:35.871 ERROR:teuthology.run_tasks:Manager failed: internal.base Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 48, in base yield File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 139, in task forced = umount_all() File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/kclient.py", line 119, in umount_all if mount.is_mounted(): File "/home/teuthworker/src/git.ceph.com_ceph_f73f1dfd96849300c4322bd7259a3ab37f94c200/qa/tasks/cephfs/mount.py", line 220, in is_mounted file = self.client_remote.read_file('/proc/self/mounts',stdout=StringIO()) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 254, in read_file proc = self.run(args=args, stdout=stdout, stderr=StringIO(), File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 450, in run r.execute() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/run.py", line 100, in execute self.client.exec_command(self.command) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/client.py", line 560, in exec_command chan = self._transport.open_session(timeout=timeout) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 959, in open_session return self.open_channel( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 1090, in open_channel raise e File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/transport.py", line 2159, in run ptype, m = self.packetizer.read_message() File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 463, in read_message header = self.read_all(self.__block_size_in, check_rekey=True) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/paramiko/packet.py", line 308, in read_all x = self.__socket.recv(n) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/virtualenv/lib/python3.8/site-packages/gevent/_socketcommon.py", line 660, in recv return self._sock.recv(*args) OSError: [Errno 113] No route to host During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/run_tasks.py", line 154, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/task/internal/__init__.py", line 54, in base ctx.cluster.run( File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in run procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/cluster.py", line 85, in procs = [remote.run(**kwargs, wait=_wait) for remote in remotes] File "/home/teuthworker/src/git.ceph.com_teuthology_e691533f9cbb33d85b2187bba20d7102f098636d/teuthology/orchestra/remote.py", line 522, in run raise ConnectionError(f'Failed to reconnect to {self.shortname}') ConnectionError: Failed to reconnect to smithi105 2024-03-24T12:01:35.887 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-03-24T12:01:35.901 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-03-24T12:01:35.915 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: OSError: [Errno 113] No route to host 2024-03-24T12:01:35.915 INFO:teuthology.nuke:Checking targets against current locks 2024-03-24T12:01:35.946 DEBUG:teuthology.nuke:shortname: smithi100 2024-03-24T12:01:35.947 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:01:35.973 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi100.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:53:40.056296', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBExVPUXHvqJ01g77jhqC9m8JF9H7OwrJ59FB7xuWe9abrrCuVBWbInMLNUiFeGmZXPIwC418IuxLihAZkbqS1hs='} 2024-03-24T12:01:35.977 DEBUG:teuthology.nuke:shortname: smithi105 2024-03-24T12:01:35.978 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-03-24T12:01:35.999 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-03-22_21:08:02-orch-squid-distro-default-smithi/7617869', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-03-24 10:53:40.054950', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBGXhGVBiLZy1w5q+/zMisSsFsc4cxVZ425x5JXIOp1EGE9jZgKW+OQIkf342Y8stnnUL9Mtjj+BBtD5OUguyDhc='} 2024-03-24T12:01:36.032 INFO:teuthology.orchestra.console.smithi100:Power off 2024-03-24T12:01:36.032 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:01:36.060 DEBUG:teuthology.orchestra.console.smithi100:power off output: Chassis Power Control: Down/Off 2024-03-24T12:01:36.061 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:36.076 INFO:teuthology.orchestra.console.smithi105:Power off 2024-03-24T12:01:36.077 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-03-24T12:01:36.086 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-03-24T12:01:36.099 DEBUG:teuthology.orchestra.console.smithi105:power off output: Chassis Power Control: Down/Off 2024-03-24T12:01:36.099 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:36.122 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-03-24T12:01:40.088 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:40.123 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:40.199 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-03-24T12:01:40.233 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is on 2024-03-24T12:01:44.202 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:44.234 DEBUG:teuthology.orchestra.console.smithi105:pexpect command: ipmitool -H smithi105.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:44.313 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is on 2024-03-24T12:01:44.343 DEBUG:teuthology.orchestra.console.smithi105:check power output: Chassis Power is off 2024-03-24T12:01:44.444 INFO:teuthology.orchestra.console.smithi105:Power off completed 2024-03-24T12:01:48.318 DEBUG:teuthology.orchestra.console.smithi100:pexpect command: ipmitool -H smithi100.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-03-24T12:01:48.429 DEBUG:teuthology.orchestra.console.smithi100:check power output: Chassis Power is off 2024-03-24T12:01:48.530 INFO:teuthology.orchestra.console.smithi100:Power off completed 2024-03-24T12:01:48.636 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/yes overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/yes 4-verify} 2-client 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 3409.4015419483185 failure_reason: 'Command failed on smithi100 with status 1: ''sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid e2f9ef6c-e9ce-11ee-95cd-87774f69a715 -e sha1=db0330b1e4e2470d52b750e251e55a522b4f7d69 -- bash -c \''ceph versions | jq -e \''"\''"\''.mgr | keys\''"\''"\'' | grep $sha1\''''' flavor: default owner: scheduled_teuthology@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=c81b87511b024b4db8e1b00438d9f473 status: fail success: false 2024-03-24T12:01:48.636 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-03-24T12:01:48.721 INFO:teuthology.run:FAIL